Results 1 to 2 of 2

Thread: Suggestions to make the WebAPI more reliable.

  1. #1
    Basic Member
    Join Date
    Oct 2011
    Posts
    75

    Suggestions to make the WebAPI more reliable.

    In this thread we shall discuss ideas on how to make the WebAPI more reliable and how to reduce load on the servers. This should be beneficial for both sides, Valve and the users.

    An idea that "I" came up with after reading several other suggestions and discussions in the old "WebAPI" thread.
    • Limit the age of matches that can be fetched to a specific time period. E.g. 4 weeks (01.10.2012 to 28.10.2012, 29.10.2012 - 19.11.12)
    • Provide dumps of older matches through a P2P platform such as BitTorrent. (Idea discussed in this thread.)
    • Limit API calls per key/per unit of time. E.g. every 5 minutes to make sure nobody kills the servers by firing thousands of requests at once.
    • Extend API to allow batch download of periods (from "last call"/specified timestamp to now/specified timestamp). This of course should have a limit to how old the first fetched match should be. Something like X*API Limit, obviously it should not extend the suggest 4 weeks. One hour or two might be a good start point.

  2. #2
    Basic Member MuppetMaster42's Avatar
    Join Date
    Nov 2011
    Location
    Australia
    Posts
    585
    Quote Originally Posted by unkie View Post
    • Limit the age of matches that can be fetched to a specific time period. E.g. 4 weeks (01.10.2012 to 28.10.2012, 29.10.2012 - 19.11.12)
    This is a poor solution.
    what happens if you as a developer, say:
    A) Don't have enough storage/memory to store every single game in a database (or hold onto a DB dump)
    B) Cannot afford to run a server to store on and thus relied on the API server to get full match details.

    I.e. an iOS/Android app that simply queries the server to look at a list of matches or a selected match's details (so you don't save anything apart from short-term caching).
    If they limited the age of matches then it would mean that this type of application would not be useful as the app would only be able to look until the age-limit imposed

    Also; even if you limit the age of the stored data, people would still try and query the database for these matches (meaning the server still has to receive and process requests, just doesn't respond the same), ergo this really creates more problems than it solves.




    Quote Originally Posted by unkie View Post
    • Extend API to allow batch download of periods (from "last call"/specified timestamp to now/specified timestamp). This of course should have a limit to how old the first fetched match should be. Something like X*API Limit, obviously it should not extend the suggest 4 weeks. One hour or two might be a good start point.
    This idea is a bit complicated to use and setup (too dynamic).
    What they really need to do is implement two new features for GetMatchHistory:
    1) MATCH_ID only (maybe call the option concise={0 or 1})
    • This feature only returns a list of match IDs (rather than players as well)
    • Most people only use GetMatchHistory to get the match IDs (and lobby type) anyways, so there is a lot of extra data that could (optionally) be cut from the
    • By removing the 10 player data slots (account_id, player_slot and hero_id), that extra data space could be in stead be used for extra match IDs in the list (i.e. return 50 in stead of 25) (thus leading to fewer calls to GetMatchHistory).

    2) Filtering by lobby type (matchmaking/co-op bot) (maybe call the option lobby_type={0 for mm, or 4 for bot})
    • If the above feature is added, then the only way to tell what lobby type a match_id is would be by GetMatchDetails-ing it, which ultimately increases the number calls rather than reducing them
    • By allowing filtering by bot/mm games, that would firstly reduce the number of useless results sent to people (because stats from co-op bot matches are effectively useless anyways......... that's another issue though),
      meaning that they get more relevant results per GetMatchHistory page, thus there are fewer pages to look through (thus leading to fewer calls to GetMatchHistory)
      but also would mean that people could use concise mode and not have to fetch details to check for bot matches (thus leading to fewer calls to GetMatchDetails)





    Quote Originally Posted by unkie View Post
    • Limit API calls per key/per unit of time. E.g. every 5 minutes to make sure nobody kills the servers by firing thousands of requests at once.
    If they implement this they have to be careful about how much is the "Limit".
    On the old thread you suggested 1 request per 5 secs, which is way too slow...
    In his very first post about the API, zoid said to "limit yourself to 1 per second". this is more than slow enough (i.e. about 25secs/page of results - maybe even still a bit slow?)

    the problem is that the devs left it up to the users to implement the limit; so some people saw it as a consequence-less choice, and thought it was fine to pull results as fast as possible (and i assume this is why the servers got so clogged up).
    They need to implement some form of limit to rectify this, if you don't limit it there will be those that take advantage of it. (i.e. there is no real choice as to this "solution")




    Quote Originally Posted by unkie View Post
    • Provide dumps of older matches through a P2P platform such as BitTorrent. (Idea discussed in this thread.)
    Great idea - people asked for this as soon as Zoid asked for feedback. I can see no real downside to this (apart from the inital getting the files out to the community) as the people that would maily download them would be the people that are copying the stats of every single MM game from the API (which I feel was a huge portion of the API calls)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •