Results 1 to 7 of 7

Thread: Bots learning from replay files?

  1. #1
    Basic Member
    Join Date
    Jan 2017
    Posts
    6

    Bots learning from replay files?

    Hello guys, I am not sure if this is the right place to post this question, but I am wondering if it is possible to let bots learn from replay files. More specifically, is it possible to access the in game state of the player's view in a replay file with APIs similar to bot APIs? For example, for each frame of the replay file, could we access:

    1. hp of visible lane creeps
    2. movement of visible enemy heroes
    3. action of current player

  2. #2
    Basic Member
    Join Date
    Dec 2016
    Posts
    76
    You may need this https://github.com/skadistats/clarity

    But for my experience, output data to console is enough for machine learning.
    I've got 600,000 ticks of data for my machine learning project.

    By the way,could you tell me what machine learning algorithm are you going to use for what purpose?
    I'm just curious
    https://github.com/lenLRX/Dota2_DPPO_bots ----My ML bot work in progress

  3. #3
    Basic Member
    Join Date
    Jan 2017
    Posts
    6
    Thanks for the reply lenlrx! I am still brainstorming the machine learning ideas so I do not have a clear picture in mind, as my plan would subject to change based on what data is available. However I do want to try out different reinforcement learning algorithms to train bots to play well. The purpose of the replay files is for supervised learning before I try out reinforcement learning algorithms.

    As for clarity, I have had a fair amount of experience with it before, but I doubt if clarity is good enough to capture the game mechanics (such as what is visible to a player).

    By the way, could you tell me how you redirect console output to your machine learning project?

  4. #4
    Basic Member
    Join Date
    Dec 2016
    Posts
    76
    Quote Originally Posted by 3kmmr View Post
    Thanks for the reply lenlrx! I am still brainstorming the machine learning ideas so I do not have a clear picture in mind, as my plan would subject to change based on what data is available. However I do want to try out different reinforcement learning algorithms to train bots to play well. The purpose of the replay files is for supervised learning before I try out reinforcement learning algorithms.

    As for clarity, I have had a fair amount of experience with it before, but I doubt if clarity is good enough to capture the game mechanics (such as what is visible to a player).

    By the way, could you tell me how you redirect console output to your machine learning project?

    use vconsole,right click and save log.
    I'm using Q learning,but I am not getting very good result.
    https://github.com/lenLRX/Dota2_DPPO_bots ----My ML bot work in progress

  5. #5
    lenlrx, I actually saw your post the other day, but forgot to leave a comment: I think it is unlikely that a MDP (Markov decision process) based algorithm (like Q learning) alone works for switching between states (you may be able to use its value for your decision making tho). There are two main problems:

    1. MDPs usually only give you the probability of changing the state (even tho the updating rules may be different), so if you think about it, it doesn't make that much sense to dota bots (for example deciding to retreat with x probability is a bit weird). you can use the numbers with some thresholds (instead of treating them like probabilities) that you define to somewhat fix this, but I'm not sure even this will be decent at the end.

    2. One way you may be able to fix the last problem is by defining many sub-states (for example defining states so that something like laning_highhp_highmana_2enemies_1tower is a valid state). But the problem with doing this is that your transition matrix can become huge (and will be huge if you actually want to define all the relevant states). Even if the matrix doesn't create memory/running time issues, learning/updating the matrix to reach some acceptable values can become insanely long.

    I haven't tried this (and haven't read your code), so let me know if you are doing something different or you think I'm wrong about something. I suggest if you want to try to improve your bots, try the second one (but not defining too many sub-states) and see what happens.

  6. #6
    Basic Member
    Join Date
    Dec 2016
    Posts
    76
    Quote Originally Posted by Platinum_dota2 View Post
    lenlrx, I actually saw your post the other day, but forgot to leave a comment: I think it is unlikely that a MDP (Markov decision process) based algorithm (like Q learning) alone works for switching between states (you may be able to use its value for your decision making tho). There are two main problems:

    1. MDPs usually only give you the probability of changing the state (even tho the updating rules may be different), so if you think about it, it doesn't make that much sense to dota bots (for example deciding to retreat with x probability is a bit weird). you can use the numbers with some thresholds (instead of treating them like probabilities) that you define to somewhat fix this, but I'm not sure even this will be decent at the end.

    2. One way you may be able to fix the last problem is by defining many sub-states (for example defining states so that something like laning_highhp_highmana_2enemies_1tower is a valid state). But the problem with doing this is that your transition matrix can become huge (and will be huge if you actually want to define all the relevant states). Even if the matrix doesn't create memory/running time issues, learning/updating the matrix to reach some acceptable values can become insanely long.

    I haven't tried this (and haven't read your code), so let me know if you are doing something different or you think I'm wrong about something. I suggest if you want to try to improve your bots, try the second one (but not defining too many sub-states) and see what happens.

    I am keep working on Q learning, because I saw a paper from facebook of SC2 using Q learning.
    I did get some result of the Q learning, and i am trying all trick i can image to improve the result.

    My mind is still not clear now,I need to read more paper's about it, If you get any idea please share with us
    https://github.com/lenLRX/Dota2_DPPO_bots ----My ML bot work in progress

  7. #7
    Basic Member
    Join Date
    Jan 2017
    Posts
    6
    The following questions might deviate a little bit from the previous discussion, but I think they would be helpful for setting up or speeding up training.

    1. Is there a way to automatically create another bot match when the current match is finished?

    2. Is there a way to end the bot match early (with script)?

    3. Is it possible to redirect or save console outputs with script only?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •