Announcement

Collapse

Forum Rules

  • No flaming or derogatory remarks, directly or through insinuation.
  • No discussion, sharing or referencing illegal software such as hacks, keygen, cracks and pirated software.
  • No offensive contents, including but not limited to, racism, gore or pornography.
  • No excessive spam/meme, i.e. copious one liners in a short period of time, typing with all caps or posting meme responses (text/image).
  • No trolling, including but not limited to, flame incitation, user provocation or false information distribution.
  • No link spamming or signature advertisements for content not specific to Dota 2.
  • No Dota 2 key requests, sell, trade etc.
  • You may not create multiple accounts for any purpose, including ban evasion, unless expressly permitted by a moderator.

  • Please search before posting. One thread per issue. Do not create another thread if there is an existing one already.
  • Before posting anything, make sure you check out all sticky threads (e.g., this). Do not create new threads about closed ones.
  • It is extremely important that you post in correct forum section.

  • Balance discussion only in Misc.
  • All art related (such as hero model) feedbacks go to Art Feedback Forum.
  • All matchmaking feedback should go here: Matchmaking Feedback
  • All report/low priority issues should go here: Commend/Report/Ban Feedback
  • No specific workshop item feedback. These should go to workshop page of that item.
  • When posting in non-bugs section (such as this), use [Bugs], [Discussion] or [Suggestion] prefix in your thread name.



In case you object some action by a moderator, please contact him directly through PM and explain your concerns politely. If you are still unable to resolve the issue, contact an administrator. Do not drag these issues in public.



All rules are meant to augment common sense, please use them when not conflicted with aforementioned policies.
See more
See less

Dota 2 Demo File Format

Collapse
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Was anyone able to find the skill build order using the Valve replay parser? Guess it's kind of a moot point since we can now pull that info from the API. I'm just curious where it was and if there's anything related to that section of the parser that might also be useful.

    Also: it's great that the patch notes had a note in the update mentioning the API improvements. When the replay parser gets updated, would it be possible to mention the parser updates in the patch notes or maybe in this thread?
    Last edited by Razumov; 02-01-2013, 11:35 AM.
    datdota.com -- Dota 2 Stats for the Professional Scene

    Comment


    • Not much has been said on the topic here for quite a while, but I just wanted to throw this out there. I'm working on a clojure library for parsing the demo files. It'll be LGPL2 and I'll stick it up on github once I have a basic skeleton in place. Don't know if there are others out there who are working on any similar projects (haven't seen any posts lately), but I'll be hanging out in the completely empty irc.quakenet.org/#dota2replay room while I'm working on it

      Comment


      • I am currently working with the Cognitive Science Lab at SFU, a group that has performed studies on the acquisition of attentional skills and expertise in StarCraft 2 (SkillCraft) , and we are very interested in taking some of the research to Dota 2.

        Our approach is to gather our data through replay analysis. The availability of replays through the WebAPI and this parsing code is a great start to provide the Lab with as much data as possible. However, I am still interested in whether a few things; namely players positions and player/client input information (clicks, screen positions, the store opening etc) are available.

        I am just getting to cracking open the demoinfo2 code to play around with, but from reading this thread it is clear that these are semi-solved/unsolved problems.

        The positional information is stored within PacketEntities, but there is difficulty finding or even calculating a hero position at a given time (or even positions movement, specified in deltas)?

        I have looked the CDOTAUserMsg_SpectatorPlayerClick, and am curious about how informative this is. Can we know the explicit map position (say in x, y) , the hero/player that issued the command, and of course the type (attack, spell, move)? If this is not the case, does anyone have any ideas on how this information and more (screen position on map of a player for example) might be retrieved.

        It is our intention to put some work into this parser and develop or modify one for the Lab's project, but I am just curious how accessible some of this desired information is to begin with.

        Thank you for all the help,

        -Ryan

        Comment


        • Originally posted by Ryan- View Post
          I am currently working with the Cognitive Science Lab at SFU, a group that has performed studies on the acquisition of attentional skills and expertise in StarCraft 2 (SkillCraft) , and we are very interested in taking some of the research to Dota 2.

          Our approach is to gather our data through replay analysis. The availability of replays through the WebAPI and this parsing code is a great start to provide the Lab with as much data as possible. However, I am still interested in whether a few things; namely players positions and player/client input information (clicks, screen positions, the store opening etc) are available.

          I am just getting to cracking open the demoinfo2 code to play around with, but from reading this thread it is clear that these are semi-solved/unsolved problems.

          The positional information is stored within PacketEntities, but there is difficulty finding or even calculating a hero position at a given time (or even positions movement, specified in deltas)?

          I have looked the CDOTAUserMsg_SpectatorPlayerClick, and am curious about how informative this is. Can we know the explicit map position (say in x, y) , the hero/player that issued the command, and of course the type (attack, spell, move)? If this is not the case, does anyone have any ideas on how this information and more (screen position on map of a player for example) might be retrieved.

          It is our intention to put some work into this parser and develop or modify one for the Lab's project, but I am just curious how accessible some of this desired information is to begin with.

          Thank you for all the help,

          -Ryan
          All of those are in PacketEntities (m_iCursor.0000, m_iCursor.0001, m_iShopPanel ...).

          Comment


          • Ryan, you may want to have a look at the Enhanced edition of the demo file dumper that was just released by Statsman Bruno. He also talks a little bit more about the format (largely a rehash of some of the stuff that onethirtyfive has already talked about in this thread). Write now the current belief is that getting exact player position is difficult without actually running a simulation of the game state due to the delta'd and encoded nature of the updates. I'd love to be proven wrong though

            Comment


            • Originally posted by mugsy__ View Post
              Ryan, you may want to have a look at the Enhanced edition of the demo file dumper that was just released by Statsman Bruno.
              I love the job Bruno is doing for the community but as he stated in cyborgmatt post, all of these are only tricks to detect levels/skills etc. Only way to get the real datas is the svc_PacketEntities message.

              Originally posted by mugsy__ View Post
              I'd love to be proven wrong though
              Code:
              tick(25786) type(Preserve) name(DT_DOTAGamerulesProxy)
              	DT_DOTAGamerules.m_iNetTimeOfDay: 60007
              	DT_DOTAGamerules.m_fGameTime: 682,355
              	DT_DOTAGamerules.m_iFoWFrameNumber: 9
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_NightStalker)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: 11, y: 35, z: 0 }
              	DT_DOTA_BaseNPC.m_angRotation[1]: 57,65625
              	DT_DOTA_BaseNPC.m_cellX: 118
              	DT_DOTA_BaseNPC.m_anglediff: -2
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Obsidian_Destroyer)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: -59, y: -27, z: 0 }
              	DT_DOTA_BaseNPC_Hero.m_iRecentDamage: 0
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Luna)
              	DT_DOTA_BaseNPC.m_iHealth: 990
              	DT_DOTA_BaseNPC_Hero.m_iRecentDamage: 20
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Pudge)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: 47, y: -33, z: 0 }
              	DT_DOTA_BaseNPC.m_angRotation[1]: 18,28125
              	DT_DOTA_BaseNPC.m_vecOrigin[2]: -13
              	DT_DOTA_BaseNPC.m_flMana: 82,01001
              	DT_DOTA_BaseNPC.m_anglediff: -23
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_VengefulSpirit)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: -4, y: -10, z: 0 }
              	DT_DOTA_BaseNPC.m_angRotation[1]: 247,5
              	DT_DOTA_BaseNPC.m_cellX: 115
              	DT_DOTA_BaseNPC.m_cellY: 120
              	DT_DOTA_BaseNPC.m_anglediff: 13
              	DT_DOTA_BaseNPC_Hero.m_iRecentDamage: 0
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Shadow_Demon)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: 3, y: 34, z: 0 }
              	DT_DOTA_BaseNPC.m_angRotation[1]: 189,8438
              	DT_DOTA_BaseNPC.m_anglediff: -1
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Axe)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: -16, y: 57, z: 0 }
              	DT_DOTA_BaseNPC.m_angRotation[1]: 70,3125
              	DT_DOTA_BaseNPC.m_iHealth: 1418
              	DT_DOTA_BaseNPC.m_anglediff: -3
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Mirana)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: 59, y: -15, z: 0 }
              tick(25786) type(Preserve) name(DT_DOTA_Unit_Hero_Lina)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: -15, y: 10, z: 0 }
              	DT_DOTA_BaseNPC.m_vecOrigin[2]: 34
              tick(25786) type(Preserve) name(DT_DOTA_BaseNPC_Creep_Lane)
              	DT_DOTA_BaseNPC.m_vecOrigin: { x: 62, y: 37, z: 0 }

              Comment


              • Are those positions part of the full state update? It was my understanding that the updates between the full state updates was where it got tricky.

                Comment


                • Originally posted by mugsy__ View Post
                  Are those positions part of the full state update? It was my understanding that the updates between the full state updates was where it got tricky.
                  Nop, this dump is made on svc_PacketEntities without using DEM_FullPacket. And YES, svc_PacketEntities is really tricky

                  Comment


                  • Thanks for the quick response guys!

                    I guess I should I have mentioned that I've looked into and played around with Bruno's parser, and it was helpful in clarifying what papa_pointnet said: that all the level, damage instance, skill information is immediately accessible, but that's about it.

                    I will start looking into the PacketEntities stuff. It seems like you've made alot of progress on that papa.
                    Last edited by Ryan-; 02-11-2013, 04:05 PM.

                    Comment


                    • Ryan,

                      If I figure it out, I will immediately document my discoveries and publish source code. Workin' on it.

                      Cheers.

                      Comment


                      • Originally posted by onethirtyfive View Post
                        Ryan,

                        If I figure it out, I will immediately document my discoveries and publish source code. Workin' on it.

                        Cheers.
                        Awesome! What are the barriers to figuring it out? Do you think it's possible?
                        datdota.com -- Dota 2 Stats for the Professional Scene

                        Comment


                        • Originally posted by Razumov View Post
                          Awesome! What are the barriers to figuring it out? Do you think it's possible?
                          So when Valve released their 'demoinfo' tool last year they provided with it a protocol specification. Here's an example of what we get for free, in the form of a parsed replay. (Warning: HUGE)

                          https://gist.github.com/onethirtyfiv.../gistfile1.txt

                          The contents of this file are similar to what Valve's own demoinfo.exe dumps, because the protocol definitions are the same. If you look at it, you'll see a lot of "classes" defined at the beginning, followed by a list of packet messages. These packet messages in turn contain "service messages", the types of which I've listed for each packet. (ex. "packet entities")

                          The problem is, the "packet entities" messages contain a "data" field with no specification. So the task is to find the relationship between those tables at the beginning of the replay and the contents of these "packet entities" datablobs.

                          papa_pointnet might have some information, if he's feeling generous?

                          The parser I'm hacking together is publicly available at https://github.com/onethirtyfive/skadi

                          Comment


                          • Originally posted by papa_pointnet View Post
                            Nop, this dump is made on svc_PacketEntities without using DEM_FullPacket. And YES, svc_PacketEntities is really tricky
                            But the structure of svc_PacketEntities is the same regardless of whether it's in a normal tick or a DEM_FullPacket right? The only difference should be if the data sent is delta from a previous tick or not.

                            And how did you get that dump?

                            Comment


                            • Originally posted by jeppew View Post
                              And how did you get that dump?
                              The Magician's Oath
                              As a magician I promise never to reveal the secret of any illusion to a non-magician.

                              Comment


                              • progress (aka "on being a magician&quot

                                I would make a horrible magician.

                                Work on my parser continues. I've successfully constructed an entity hierarchy from the SendTables command. Here's the hierarchy from my demo:
                                https://gist.github.com/onethirtyfiv...531d9d2a02e6b8 (best viewed 'raw')

                                Here's what I know about each entity's properties:
                                https://gist.github.com/onethirtyfiv...a94a66d4400cb8 (best viewed 'raw')

                                I'm still unclear on what the property 'flags' mean. I took some guesses from papa_pointnet's code snippets earlier in the thread. they appear to be type hints when decoding bit streams.

                                Source code available at https://github.com/onethirtyfive/skadi

                                If anyone wants to jump in, mugsy and I are pretty much always in #dota2replay on irc.quakenet.org.

                                Cheers!

                                Comment

                                Working...
                                X