Page 1 of 2 1 2 LastLast
Results 1 to 10 of 12

Thread: LUA Memory usage warning: Ideas?

  1. #1
    Basic Member
    Join Date
    Nov 2015
    Posts
    108

    LUA Memory usage warning: Ideas?

    Anyone have any clever ways of hunting down the cause of the following warning:
    Code:
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 83,886,080 bytes.
    Edit: I found it just going through printing the entity count of all my tables. I was assuming that I missed something somewhere and am just continuously adding entries to a table and not clearing any and I was right.

    But for the future, anyone got anything clever for this one?
    Last edited by ironmano; 03-21-2017 at 03:54 PM.

  2. #2
    Basic Member
    Join Date
    Nov 2015
    Posts
    108
    Just an FYI for anyone that cares, it seems data in the package.loaded table, which is where all your modules loaded with require() are stored, is not reset when you use the dota_bot_reload_scripts command. This proves to be mighty handy tracking down any errors like I described above as long as of course they are in such a module. Using a print_r(package.loaded) statement right after my requires to print the table shows me right away what table I was inflating so bad.

    Of course you'll need a print function that will traverse the table and print it like print_r from https://coronalabs.com/blog/2014/09/...able-contents/
    And using -vconsole helps to read the sometimes massive output.

  3. #3
    Basic Member
    Join Date
    Dec 2016
    Posts
    121
    Got the same problem, may I ask which kind of table is it? Is it a table of handle?

  4. #4
    Basic Member
    Join Date
    Nov 2015
    Posts
    108
    Yeah, just a big table full of tables. There is a mountain of data in there, it even contains the _G global table as far as I can tell. It's just that everything except modules loaded with require() seem to be purged on dota_bot_reload_scripts.

  5. #5
    Basic Member
    Join Date
    Feb 2017
    Posts
    19
    Thanks ironmano for the followup posts and insight.

    I printed that package.loaded table and it's insane. Not sure how to reasonably look through it for waste.

    I did notice one thing about how a context is apparently duplicated for an order filter callback, so maybe I have a problem there. I might see if I can write something to analyze changes to that table.

    Are you saying that as the VM size grows that there's a good chance that we'll find the expanding data in that table somewhere?

    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 83,886,080 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 123,731,968 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 157,286,400 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 204,472,320 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 238,026,752 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 271,581,184 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 305,135,616 bytes.
    [ W VScript ]: LUA Memory usage warning: The VM has hit a new high usage of 370,147,328 bytes.

  6. #6
    Basic Member
    Join Date
    Mar 2012
    Posts
    1,681
    The problem is most likely due to circular references. I wouldn't be surprised if table B is referenced in table A that is referenced in table B that is referenced in table A...
    And prolly updated either in a loop or inside Think() under certain conditions. 300 MB is huge. anything over 10 is prolly overkill somewhere.

    I recommend a (re)design analysis on your code. That's how I solved most of my circular dependencies
    (I admit I am using a full takeover so it's easier since I have full control over the whole code)
    Explanations on the normal, high and very high brackets in replays: here, here & here
    Why maphacks won't work in D2: here

  7. #7
    Basic Member
    Join Date
    Nov 2015
    Posts
    108
    For my issue it was a simple comment error. I was caching all enemy hero data I wanted every cycle and it was supposed to clear the table each cycle as well but the line that cleared it got caught in a block comment with some test garbage.

    If your table is really that large it should be very easy to find when you print it, I mean really, my table was no where near that huge data-wise and was still like 80 pages in vconsole.

  8. #8
    Basic Member
    Join Date
    Feb 2017
    Posts
    19
    I guess I'm in the wrong thread -- I'm actually building a custom mod, not a bot.

    However, similar logic probably applies.


    Quote Originally Posted by The Nomad View Post
    The problem is most likely due to circular references. I wouldn't be surprised if table B is referenced in table A that is referenced in table B that is referenced in table A...
    And prolly updated either in a loop or inside Think() under certain conditions. 300 MB is huge. anything over 10 is prolly overkill somewhere.

    I recommend a (re)design analysis on your code. That's how I solved most of my circular dependencies
    (I admit I am using a full takeover so it's easier since I have full control over the whole code)
    I absolutely have circular references. Each of my creatures has a link to the spawner, and the spawner has a link to the creature. And when the creature dies, I'm not sure whether I remove the creature from the spawner's table.

    I can't imagine that particular code causes 300mb to spin out of control, though. But I probably have other things like this. I do a lot of nested contexts, which immediately jumped out at me when I started to dump the package.loaded table.

    I use nested contexts in javascript a lot, so I may be abusing this technique in Lua. Something like...

    function SomeClass:Thing(params)
    SomeOtherObject:OtherFunction(params, function(result)
    self:ActOnResult(result)
    end)
    end

    Where the idea is to provide a callback into some long-running function, or messy verbose function. However, this seems to load up the package.loaded table with duplicate copies of the "self" context.

    D'oh.

    Quote Originally Posted by ironmano View Post
    For my issue it was a simple comment error. I was caching all enemy hero data I wanted every cycle and it was supposed to clear the table each cycle as well but the line that cleared it got caught in a block comment with some test garbage.

    If your table is really that large it should be very easy to find when you print it, I mean really, my table was no where near that huge data-wise and was still like 80 pages in vconsole.
    Yes, trying to print packages.loaded goes on... forever? =)

    Really appreciate it guys. This at least gives me something to look into, rather than just some obscure warning. Would be nice though to have better tools to figure out why the VM gets to 1gb after 30-60 minutes of play.

  9. #9
    Basic Member aveyo's Avatar
    Join Date
    Aug 2012
    Location
    EU West
    Posts
    2,526
    Are you really inspecting / copy-pasting lengthy stuff from console / vconsole (horrible piece of crap)?
    That does not sound optimal at all - why not add launch options: -consolelog -conclearlog
    You could also add time stamps with: -con_timestamp
    Then inspect the resulting \Steam\steamapps\common\dota 2 beta\game\dota\console.log

    Can also filter output, see:
    Code:
    //// AVEYO: DISABLE ALL EXTRA CONSOLE AND LOG SPEW SOURCES FOR A CLEAN SCRIPT OUTPUT ( VALVE PLZ GIFF SINGLE CMD FOR IT )
    log_verbosity Console off | grep %
    log_verbosity AnimationGraph AnimationSystem AnimGraphManager AnimResource Assert "BitBuf Error" BoneSetup Client "Combat Analyzer" CommandLine D3D Decals Demo DeveloperVerbose DotaGuide DOTAHLTVCamera off
    log_verbosity DOTAHLTVDirector DOTA_CHAT DownloadManager EmitSound EngineInitialization EngineServiceManager "Entity Dump" "Entity Load Unserialize" "Entity System" Filesystem GameEventSystem GCClient GlobalState HangWatchdog "HLTV Server" Host off
    log_verbosity HostStateManager IME InputService InputSystem InstantReplay LOADING MaterialSystem MeshSystem ModelCombiner modellib NavMesh NetworkClientService Networking "Networking Reliable" NetworkP2PService NetworkServerService off
    log_verbosity NetworkService Particles ParticlesLib Physics PostProcessing PostProcessPipeline RenderPipelineDota RenderPipelineVr RenderService RenderSystem ResourceSystem SaveRestore SaveRestoreIO Scaleform "Scaleform IME" ScaleformAS off
    log_verbosity ScaleformParse ScaleformScript SceneSystem SchemaSystem SchemaSystemUtils Server ServerLog SignonState SndEmitterSystem SndOperators SoundOperatorSystem SoundOpGameSystem SoundSystem SoundSystemLowLevel SpawnGroup SplitPacket off
    log_verbosity SplitScreen Steam SteamDatagramClient SteamDatagramServer SteamUnifiedMessages ToneMapping ToolGameSimulation TypeManager Vfx VguiCallQueue VolumetricFog VProf VR WeekendTourney Workshop WorldRenderer off
    //// AVEYO: ENABLE JUST BASIC CONSOLE SPEW
    log_verbosity Developer DeveloperConsole Panel Panorama PanoramaScript VScript VScriptDbg VScriptScripts CustomUI CustomGameCache CustomNetTable detailed
    log_verbosity General essential // 'essential' is recommended for this channel, but if you need stuff like echoln, use 'default' instead
    log_verbosity Console default | grep %
    then you could re-enable just the channels that interest you.

  10. #10
    Basic Member
    Join Date
    Feb 2017
    Posts
    19
    Thanks Aveyo. I tried that, and it does indeed spit the console data out to that log file. So I can inspect larger files.

    I've created a script, which every 0.2 seconds, it walks the entire package.loaded table. It counts tables, and values. Then, it prints the diff. This has actually helped me verify that there doesn't appear to be a recursive table memory leak. Once the game loads, it stabilizes, then as I fight creatures, the values rise, then fall. Eventually normalizing back to roughly the same number of tables and values as when the game loads.

    Bummer. I was really hoping I would be able to find some massive table memory leak causing this issue. :-/

    When the game loads, it quickly gets up to ~16 megs or so. And I load a bunch of stuff into memory. However, then the table counts remain roughly the same through 20 minutes of play, but the vm size constantly increases.

    05/06/2017 - 17:53:14: [VScript] LUA Memory usage warning: The VM has hit a new high usage of 16,777,216 bytes.
    05/06/2017 - 17:53:46: [VScript] LUA Memory usage warning: The VM has hit a new high usage of 51,380,224 bytes.
    ...
    05/06/2017 - 18:44:09: [VScript] LUA Memory usage warning: The VM has hit a new high usage of 291,504,128 bytes.

    Sigh.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •