dota2_nn: Extremely WIP deep supervised learning full-takeover bots

[axetion]

https://github.com/axetion/dota2_nn

This is the start of a crude deep learning bot using Torch I've been working on. Unlike most of the previous ML bots here, this is purely supervised learning, which means that a set of examples is constructed ahead of time for the models to learn from as opposed to slowly learning in-game (unsupervised). These examples are made by extracting basic information on actions made by the top 3 players on the winning team (currently measured by kills, I intend to use fantasy Dota points instead as soon as I figure out how to extract them) from replays using Dotabuff's excellent Manta replay parser (written in Go) along with some nasty data munging of the bowels of the Source engine.

Currently installation is not very streamlined (I intend to package up the server that handles the Torch requests into a single executable as soon as the bots are ready for people to actually play against):

- Install Torch
- Install dependencies for the server:

luarocks install luajson
luarocks install uuid

luarocks install https://raw.githubusercontent.com/benglard/waffle/master/waffle-scm-1.rockspec

- Extract repo to <steam path>/steamapps/common/dota 2 beta/game/dota/scripts/vscripts/
- Start the server with

th <steam path>/steamapps/common/dota\ 2\ beta/game/dota/scripts/vscripts/server/server.lua

before starting a match

I haven't included any pre-trained models with this, since currently only moving and attacking is working (no purchasing items yet) and I don't have the replays to make good models (I tried with 40 replays, which sounds like a lot until you realize that that accounts to 1-2 matches per hero at most which is very little to generalize from) -- the best way people can contribute is by getting me more replays, since you can only download 100 replays per 24 hours. To train your own models:

- Get Go
- Clone repo (if you're doing it from the command line, be sure to use the --recursive option)
- If you're on Windows, run set "GOPATH=%cd%". Otherwise (OS X, Linux) run export GOPATH=`pwd`
- Build the corpora from the replays:

go run corpus_builder.go <replay files...>

- Train models:

th trainer.lua

- Move ability_data.lua to <steam path>/steamapps/common/dota 2 beta/game/dota/scripts/vscripts/bots
- Move the data folder to <steam path>/steamapps/common/dota 2 beta/game/dota/scripts/vscripts/server

This is still very much a WIP, but I wanted to get it out a little early to demonstrate the concept and get some feedback

[TheP1anoDentist]

Ill try and run it today. Looks cool.

Would be nice to see a sample data file. So I can see what the inputs are...and how determine output actions.

i.e. I imagine you can get attacks from combatlog damage to other things.
you could use spell usage in a similar way if thats the case, but working out where to target aoe spells would be do-able but not trivial.

I was considering just starting out doing it with a 1v1 mid bot for simplicity, but one thing thats a big part of 1v1 is drawing creep aggro and I couldnt figure an easy way to extract that information from dota2 replay data.
I think you could possibly track creep positions, see that theyve moved towards a hero despite having other enemy creeps in range...inferring they've been aggroed. but again its not simple

I imagine there's going to be so many inputs for 'playing dota in general', will need a lot of nodes, and so you'll literally need >1million games to not overfit

"you can only download 100 replays per 24 hours" dotabuff and opendota must get around this somehow. although they may have to use thousands of steam accounts to do it

[TheP1anoDentist]

I dont know what corpus_builder.go is :/

EDIT: found it in your nn_trainer.

edit: my issue was trying to use source1 replay. working now

[DreadedGhoul575]

Sounds hard to install. I hope someone uploads a video of this in action though.

[TheP1anoDentist]

Yes as I had assumed quite limited inputs.

but its still really impressive and a great starting point.

GetTargetIndex() is awesome. exactly the thing I was trying to find in clarity (the java parser) and couldnt spot.
the bots can probably learn aggro themselves with your setup. they attackmove hero then move command before attack finishes

goroutines are pretty fun to play around with. you could use them to do replays in parallel for generating the training data

Im still a NN noob. but some things:

I dont see any feature scaling. does torch handle the feature scaling automatically from the inputs, have I missed where you've done it...does it need adding?

[TheP1anoDentist]

I think the biggest issue to start with is getting enough replays.

Going to be hard to tell if models are working well, or failing. if never have enough results to train on.

Might be worth going to opendota discord channel and maybe ask if its possible to replicate their setup.

what you could do, is plan to train 5/10 heroes only to start with. just find matches with them through the web api, and only parse their replays

I could run the corpus_build.go on a couple of my servers and have the results be downloadable. could get 200 per day.

In general want to be appending your example files rather than overwriting when you re-run with new replays, or at least have the option to do so.
either that or have a separate file for each match, and the torch lua stitches them together

I think 2_moveexamples, 3_moveexamples is kind of confusing naming. dire/radiant would be much clearer. I know 2/3 comes from their name in the protobuff..but dont need the examples files named like that.

confused me for a bit.

(if you want to get 100 replays fairly quickly, whilst filtering to make sure a certain hero is in those games, can use the --downloadOnly option for a tool I have https://github.com/ThePianoDentist/dotaHighlightHelper
only works on pro-games..but that shouldnt matter
)

[TheP1anoDentist]

I would rather contribute to this, than do my own NN stuff from scratch.

When you are ready for contributions, if you have a rough plan of the multiple things that need improving/changing. It would be good to make github issues so I dont go and work on stuff you've already improved.

Would you be ok with people creating github issues for things they would like to see changed (even if just for refactoring purposes....i.e the MAMMOTH secondPass function )

[nostrademous]

What has been your result thus far?

[axetion]

Sounds hard to install. I hope someone uploads a video of this in action though.

I intend to make the server just a prebuilt statically-linked binary (so no Torch or dependencies needed) once I figure out how this works: https://github.com/ers35/luastatic. (I'd much rather not have to use the server at all and just use Torch in Dota 2 directly though...)

I think the biggest issue to start with is getting enough replays.

Going to be hard to tell if models are working well, or failing. if never have enough results to train on.

Might be worth going to opendota discord channel and maybe ask if its possible to replicate their setup.

They actually have a docker image so I could do that, however you need multiple steam accounts to be able to process the amount of replays they/Dotabuff get.

what you could do, is plan to train 5/10 heroes only to start with. just find matches with them through the web api, and only parse their replays

I think restricting the hero pool would be a good idea for starting out, especially considering right now heroes like Doom and Rubick that steal spells screw up the corpus builder anyways.... :PP

(if you want to get 100 replays fairly quickly, whilst filtering to make sure a certain hero is in those games, can use the --downloadOnly option for a tool I have https://github.com/ThePianoDentist/dotaHighlightHelper
only works on pro-games..but that shouldnt matter
)

Ooh, this looks nice. Will look into.

Yes as I had assumed quite limited inputs.

but its still really impressive and a great starting point.

I have more things to add on my checklist (status modifiers and number of nearby summoned units are the biggest ones), but the limiting factor is really retrieving them from the replays. Very undocumented spooky magic going on there.

I dont see any feature scaling. does torch handle the feature scaling automatically from the inputs, have I missed where you've done it...does it need adding?

I think you might have just missed it: here's where coordinates are scaled for instance. I am using rectified linear units though so scaling matters a little less.

Would you be ok with people creating github issues for things they would like to see changed (even if just for refactoring purposes....i.e the MAMMOTH secondPass function )

Absolutely -- pull requests welcome too! Once I get a little more organized I'll make a GH project board that I can add tasks and issues too so that people can see progress on things.

Yeah, SecondPass is nasty (understandbly though, it's the meat and potatoes of it). I'll split it up into SecondPass and ConstructFeatureVector when I get a chance.

[axetion]

GetTargetIndex() is awesome. exactly the thing I was trying to find in clarity (the java parser) and couldnt spot.

Here's the full proto: https://github.com/dotabuff/manta/bl...ges.proto#L562

To be honest I consider myself lucky that I found it :P

[axetion]

What has been your result thus far?

The models trained with 40 random replays (which again amounts to only 1 replay per hero usually) moved around (spotted them standing in ward spots and following the creep front) and knew when to run away but didn't really use abilities. I've weighted the training towards learning abilities appropriately so I'm pretty sure I need more examples for it to infer when to use them. (Another possibility is that it might just be thrown off by the fact that doesn't have any items.)

[nostrademous]

A better installation approach would be really useful. I just tried to install it (per directions) and ended up getting a critical error when doing the "luarocks install json".

It's kind of strange though, b/c I install torch (Windows 10) per their instructions (with Anaconda3) first in order to get luarocks in the first place, but the message I got is:

json 1.0-0 depends on torch >= 7.0 (not installed)

So it's probably a PATH env issue.

[nostrademous]

(Another possibility is that it might just be thrown off by the fact that doesn't have any items.)

Easy way to test this... don't gut the item_purchase_generic

[axetion]

A better installation approach would be really useful. I just tried to install it (per directions) and ended up getting a critical error when doing the "luarocks install json".

It's kind of strange though, b/c I install torch (Windows 10) per their instructions (with Anaconda3) first in order to get luarocks in the first place, but the message I got is:

json 1.0-0 depends on torch >= 7.0 (not installed)

So it's probably a PATH env issue.

Actually that was my fault -- I gave the wrong package name. The correct one is luajson and I've updated the post.

(Although I would recommend checking to see if the luarocks you're using corresponds to Torch's luajit not whatever other Lua install you might have)

[nostrademous]

Actually that was my fault -- I gave the wrong package name. The correct one is luajson and I've updated the post.

(Although I would recommend checking to see if the luarocks you're using corresponds to Torch's luajit not whatever other Lua install you might have)

I'm running it out of the torch\install directory in an Anaconda shell.

installing luajson works no problem now.

installing pegasus throws an error (seems lzlib dependency fails):

(C:\Users\nostr\Anaconda3) C:\torch\distro-master\install>luarocks install pegasus
Installing https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/pegasus-0.9.2-1.src.rock
Missing dependencies for pegasus 0.9.2-1:
   lzlib >= 0.4.1.53-1 (not installed)

pegasus 0.9.2-1 depends on lzlib >= 0.4.1.53-1 (not installed)
Installing https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/lzlib-0.4.1.53-1.src.rock

Error: Failed installing dependency: https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/lzlib-0.4.1.53-1.src.rock - Could not find header file for ZLIB
  No file zlib.h in c:/external/include
You may have to install ZLIB in your system and/or pass ZLIB_DIR or ZLIB_INCDIR to the luarocks command.
Example: luarocks install lzlib ZLIB_DIR=/usr/local

Digging some more, looks like lzlib expects you to pull down the zlib source and build it first and just point the luarocks include location at it (not sure why it doesn't just do it for you...)