Giter Club home page Giter Club logo

swgohapi's Introduction

swgohapi

Cached, parsed https://swgoh.gg/ player profile data

WARNING this is a beta software, under active development. Use at your own risk.

Using the public version

Tired to scrape the website constantly for data consumption? Getting too much HTTP 402 responses? Freat not, here is your solution.

This API is designed to be an easy escrape-to-JSON converter. Using ronoaldo/swgoh as parsing library, this microservice caches the parsed data for it's lifetime (around 24hs) and serves responses in about 80ms.

Leaveraging memcache for ultra-fast responses, and with a on-demand, background syncing engine that does not overloads the website, this API can be used by apps and bots to enhance the player experience.

Please note that you may get blacklisted, graylisted and all sorts of "listed" things if you use this to bad things.

As uncle Old Ben once told us: "with great power comes great responsibility"

Hosting your own version

It requires a Google Cloud Platform account, but depending on your usage, you should stay in the free quota. You also will need to install Google Cloud SDK and Google App Engine Go SDK.

appcfg.py update --application your-project-id --version beta .

Please refer to the Google Cloud Platform documentation on Google App Engine and setting up gcloud and the SDK.

swgohapi's People

Contributors

ronoaldo avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

swgohapi's Issues

Use memcache

Use memcache to avoid hitting datastore and to make operations faster.

Convert synchronous to asynchronous profile updates

Now that profiles are always on cache, instead of making user-facing requests take longer time, the API behavior should be:

  1. For a never-seen profile, instead of fetch basic info, return a 202 Accepted status code, and schedule the full sync in background task queue. Return the estimated time the result will be ready (can be hard coded as of now to 5s).
  2. For cached profiles, if cache is expired (i.e., < 24hs), schedule a full sync in background with a named task to avoid overload.

Now that we have a periodic check of cached profiles every 6hrs, we can rely on that periodic updates to provide properly updated info, and make response-times of user-facing request a constant (<100ms).

For low arena ranks, no arena history or info is shown and profiles never get loaded

This is a sample, valid profile page that never gets loaded up: https://swgoh.gg/u/miamizalandas/

From logs, it failed to load the arena team, but there is no arena team!

Reloading profile (fullUpdate=true)
Loading profile miamizalandas
Not found on memcache, fetching from datastore (err=memcache: cache miss)
Loading arena team ...
Unable to reload profile: swgohgg: profile miamizalandas may not have Commander Luke Skywalker activated. (err=swgohgg: unable to find collection for profile 'miamizalandas')

Rework stats and add ship stats too

It may be a good idea to merge the Collection and Stats fields and to expand the Ship collections with the unit stats extracted. This way the Profile.Collection[0] contains all info at once. This requires a rework on the github.com/ronoaldo/swgoh/swgohgg package to merge the same info.

One option is to make the a new Character.Stats and Ship.Stats attribute to hold the info, and make it optional in the response JSON.

Add /v1/guild/guild-id-or-name endpoint

Return basic stats for all profiles that belong to a given Guild.

GET on the guild endpoint could return all 50 profiles in that guild. A new Guild field could be added to make this request faster.

GET on the guild endpoint, if there are no profiles in that guild, could then fetch the guild page, parse all profiles in there and schedule a profile sync for each one.

Add batch/filter options to /v1/profile

The current profile endpoint, /v1/profile/username, could be reworked to allow batch get / filter when no username is specified.

New /v1/profile/ endpoint that can return a list of profiles with the filter criteria via query string values:

  • ?profile=user1&profile=user2...&profile=user3: return the specified profiles back.
  • ?guild=guild1: return profiles from the specified guild name.

If no query string values are specified, return an empty result.

To some extent, this proposal returns an unidimensional array equivalent of the current api for the first filter when only one profile is set.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.