Page 31 of 82 FirstFirst ... 21 29 30 31 32 33 41 81 ... LastLast
Results 301 to 310 of 814

Thread: Dota 2 Match History WebAPI

  1. #301
    Quote Originally Posted by walkingcarpet View Post
    My crawler has been getting large amounts of empty results on queries, and then suddenly maxes out.
    [snip]
    A difference of merely one second and yet returns over 500 results...
    Quote Originally Posted by Zoid View Post
    The change today made it so date ranges must be at least 24 hours apart, as individual seconds was causing excessive load on the database server.
    Just in case there was any confusion, my method for crawling did not involve querying every one second. It dynamically adjusted the date range for optimal crawling in a simple manner. Basically it tried a date range and nudged it up or down based on how many results were received.

    Where date_range is a number of seconds representing a time span (such as two hours), and "shift" is to adjust search range where:
    1. set date_max = last_date_min
    2. set date_min = date_max - date_range


    Logic to capture every match ID would then be:
    1. set date_max = now()
    2. set date_min = now() - date_range
    3. query GetMatchHistory for results
    4. too many results (total_results >= 500): do not shift, but decrease date_range by 10% then redo the query. for example 2 hours would become 1.8 hours
    5. no results (total_results = 0): shift and increase date_range by 10% then redo the query. for example 2 hours would become 2.2 hours
    6. good amount of results (400 <= total_results < 500): this was the ideal case. iterate through results using start_at_match_id, then shift but keep the same date_range value, then query for next results
    7. results but not optimal (0 < total_results < 400): iterate through results, shift and increase date_range by 10% (to hopefully get between 400 and 500 on the next result set) then query for next results


    The only reason it was attempting a 1 second date range was due to the crawler logic for adjusting the date_range prior to the API changes on Aug 8th/9th

    I'm just clarifying that I was attempting my best to responsibly crawl the API and was not performing a query for every 1 second range.
    Last edited by walkingcarpet; 08-11-2012 at 01:38 PM.

  2. #302
    Basic Member
    Join Date
    Jul 2012
    Posts
    11
    I have a proposition of *simple solution* that may resolve a lots of problems

    Im thinking about creating a file containing all match id's for particular day in json format.

    The filename format:

    YYYYMMDD.json

    So if i want all match id's for 5 feb 2011 im just downloading file:

    20110205.json

    and then i just asking web API for match details.

    Or even better - file may contain not just match id's but details of the matches, so i don't even need to do single API call to valve server.

    PROS:
    - a much less number of requests to valve servers, much less load on database server
    - i think dumping a one day data is easy to implement

    CONS:
    - 24 hour lag as there is not file for today but i think it would be acceptable by most peoples (?)

    Please, please, please - consider it seriously.

  3. #303
    Basic Member Shinkai's Avatar
    Join Date
    May 2012
    Posts
    186
    That would make it waaaaaaay much easier for us to get data instead of going as we have to go now. Definitely thumbs up for that suggestion

  4. #304
    Basic Member
    Join Date
    Jul 2012
    Posts
    11
    If the 24 hour lag is too much, it always can be 1 file for one hour (so the lag is one hour only).

    It would produce 8760 files per regular year (365 days) which is not very much.

    But there is one potential problem: there can be a match that started near end of the hour and finished much later
    (i've seen games that took about 90 mins).

    So the file may be incomplete after it's created, or contain id of match not yet finished.
    Last edited by BlueCloud; 08-11-2012 at 08:10 PM.

  5. #305
    Basic Member
    Join Date
    May 2012
    Posts
    15
    Anyone can thest the replay url?

    http://replay<cluster>.valve.net/570/<matchId>_<replay_salt>.dem.bz2

    Its not working here.

    Another thing, how can this be possible:

    https://api.steampowered.com/IDOTA2Match_570/GetMatchDetails/V001/?match_id=24377568&key=<key>&format=json

    This match is lobby_type=0, and human_players = 9, how, and why? hehe
    Last edited by hor; 08-12-2012 at 06:50 AM.

  6. #306
    Basic Member
    Join Date
    Apr 2012
    Posts
    39
    @Zoid or anyone, could you give me the first match_id for each month (I'd rather not write a binary search using your API ;3).

    Thanks in advance.

  7. #307
    Basic Member
    Join Date
    Aug 2012
    Posts
    5
    nice fix of performance before fix i do about 100 request per second for getting game details, after fix web api blocks for approximately 15 seconds my request after 1000 requests, maybe worth describe this feature ?

    why do not cache answers of played games ? they do not change in time
    Last edited by dessoya; 08-12-2012 at 11:57 PM.

  8. #308
    Basic Member
    Join Date
    Jun 2012
    Posts
    18
    Hey,

    I wanted to do error handling for when the web api is down. Basically I just ask for the first 25 matches using the basic query, and if it returns 0, i handle the error saying the web api is down. I wonder if there's a better way to do this (check server status, etc)? I think it would be nice to have a service that essentially returns server status

  9. #309
    Basic Member
    Join Date
    Aug 2012
    Posts
    5
    I have a suggestion to make command for getting 1000 or 10000 games per request, this will reduce the load of scaning for stat sites.

  10. #310
    Quote Originally Posted by Phantasmal View Post
    I've hit something weird that I don't quite know how to explain.

    https://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001/?date_max=1344423600&skill=2&key=<key>

    has as it's most recent entry

    "match_id": 31287964,
    "start_time": 1344383953,

    I can get a more recent entry using

    https://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001/?date_max=1344470400&skill=2&key=<>

    which returns new entries taking place between 1344470349 to 1344469997, but

    https://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001/?date_max=1344470399&skill=2&key=<>

    brings up the same list as the first search.

    After figuring that much out, I punched in the UTC codes to a converter. Match 31287964 took place 47 seconds before August 8th, 12:00 AM GMT. The lowest date_max search that returns something other than that match is set to August 9th, 12:00 AM GMT. Basically any date_max set anywhere inside of August 8th becomes August 7th, 11:59 PM.

    Maybe I'm doing something dumb to trigger this that I haven't noticed. Maybe it's just related to the downtime. Whatever the case, here's a heads-up for anyone running into a similar issue.

    Edit: Turns out that it's a little more complicated than that. I run into the same issue no matter what day I start on. If I've used date_max recently, future searches seem default to the previous day. Don't know if this is a side effect related to the date_min, date_max change or if I'm specifically breaking something.

    Edit2: Recent searches don't matter. I can pick any time (GMT) in any day. Use a date_max search with it. The first result will always be shortly before midnight of the previous day.

    Edit3: So date_min and date_max only function if you're explicitly defining a range that is at least a day and the total search has less than 500 entries.

    Also, if you set your date_max to July 31st 12:00 PM (GMT) and your date_min to July 30th 12:00 PM (GMT) your first result will be no later than July 30th 11:59 PM (GMT) even when there are less than 500 entries. If you want to get matches that happened on July 31st your date_max has to be any time on August 1st (and there have to be less than 500 matches ahead of it in that particular search).
    Quote Originally Posted by MuppetMaster42 View Post
    I am +1-ing the date glitch.

    right now,
    if I use this url: https://api.steampowered.com/IDOTA2M...ed=25&key=XXXX
    then the latest match is ID#32124620 from time 1344694719

    however if I use this url: https://api.steampowered.com/IDOTA2M...max=1344698029
    then the latest match is ID#31886792 from time 1344611006
    Any update on this issue? I am also having issues with using the min and max date.

    What it seems to be doing is only retrieving matches from right before midnight of day (X - 1) if the date_max's timestamp falls in day (X) before the midnight of day (X).

    This issue makes it impossible to retrieve any matches prior to the 500 matches preceding the midnight of day (X) if one is using date_max, since one can set the timestamp of date_max to a timestamp that falls within day (X + 1) or just the midnight of day (X). Any value less and it will retrieve day (X - 1)'s matches.

    Edit 2: After some further testing, removed old post data and updated with confirmed data.
    Last edited by ex_eternity; 08-13-2012 at 09:23 AM. Reason: Edit 2

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •