Giter Club home page Giter Club logo

splits-io's Introduction

Splits.io

View performance data on Skylight View performance data on Skylight View performance data on Skylight License


A speedrun data store, analysis engine, and racing platform.

About

Splits.io is how speedrunners improve through data. It gives split-by-split analysis of individual runs, viewed through a lens of all runs. On Splits.io, speedrunners share more than their time—they share their entire history of attempts, successful or not, and get feedback on how to improve long-term through statistics and comparisons with themselves and other runners in their weight class, both live (via races) and after-the-fact using historical data.

Splits.io works with LiveSplit and more than 15 other speedrunning timers. An auto-generated list can be viewed in the FAQ; new timers can self-integrate using the Splits.io Exchange Format.

API

Client libraries exist for the following languages. These are created and maintained by community members:

For full API documentation and when using other languages, see the API readme.

Local Development

Splits.io runs on Docker, which makes it easy and consistent to set up and run on any machine despite any unusual dependencies. The one downside is that you must first install Docker!

Requirements

Special note for Windows

Because of how Splits.io uses Docker, Windows requires WSL2 to be installed to run Splits.io. If you haven't done so already, follow these instructions when running Splits.io on Windows:

  1. Install Windows Subsystem for Linux (WSL) on Windows 10
  2. Docker Desktop WSL 2 backend

These steps are not required for Linux or macOS.

Running

First run

The first time you run Splits.io will take a while to build, so you should go grab a coffee or something:

make  # shorthand for make build run

Once the output looks settled (you should see * Listening on tcp://0.0.0.0:3000), you're good to go! Access localhost:3000 in your browser. The first page load after a new build may also take a minute.

Note: If the page has no styling information on first run, try rebooting the server.

Now that Splits.io is running, one last step you should perform in another terminal window is:

make seed

This is required, as it sets up a client ID/secret for your local JavaScript to communicate with your local backend. It will also prepopulate your local database with some initial data.

Subsequent runs

If you have already run make or make build in the past, you usually won't need to rebuild everything again, so you can instead just run the faster version:

make run

If you do need to rebuild (e.g. when adding or upgrading a dependency) but don't necessarily want a server:

make build

Further Setup

These steps are not required for normal operation, but you may want to perform them for specific categories of work.

OAuth

Some features are built on top of links with other platforms, like Twitch sign-in. If you want these features to work, you need to register developer applications with the appropriate services. Copy .envrc.example to .envrc and follow the comments inside for details and instructions for various platforms.

After following the instructions, run

source .envrc
make build run

to rebuild the server with your new environment variables. We recommend using direnv to automate the first step whenever you change directories!

Emails

Splits.io sends emails when users go through the "I forgot my password" flow. In development mode, these emails are not actually sent but are instead generated then saved to tmp/mails.

If you want to preview a demo email in your browser, you can fiddle with the previewers in spec/mailers/previews then access a URL like

http://localhost:3000/rails/mailers/:mailer/:action

such as

http://localhost:3000/rails/mailers/password_reset_token_mailer/create_email.

Debugging

Getting Up and Running

If you're having trouble getting Splits.io running at all using the above instructions, please make a GitHub issue so we can work it out! Even if you think it's a silly issue, the fact that it's happening to you means we haven't ironed out everything (even if the only thing preventing you from setting up is better documentation!).

Working with the Code

If you have the app up and running but are looking for insight into debugging your own changes, you can access a Rails console inside the Docker container with

make console

Attaching to a debugger

If you use binding.pry anywhere in the code, once you hit the breakpoint specified use the command

make attach

in another terminal window to attach to it. To detach, make sure to exit the debug session then use the docker attach escape sequence ctrl + p then ctrl + q.

If you need to attach to a container other than web, specify a container with the syntax

make attach container=worker

Running Tests

To run tests from inside the Docker container, use

make test

To run only specific tests, use

make test path=spec/path/to/test/file/or/dir

Linting

We use Rubocop for code cleanliness and styling. To run it against changed files, commit your changes and run

make lint

Profiling

Splits.io utilizes a few libraries for profiling our code.

Rack Mini Profiler is used to find major slowdowns in the code through the use of the badge in the top left corner of the browser window. There is also a slew of different URL parameters that you can use to get more detailed information about various aspects of the request. Details of these are explained in the readme for RMP. To get more detailed information about how code will perform in a production like environment, run

make profile

to boot the app in the profiling environment, which has most of the production flags toggled on.

DerailedBenchmarks is used to test memory over lots of requests. The commands that can be run are detailed in the readme for DB. When you have a command you want to run, use the make task like so with the options that you need

make derailed env="-e TEST_COUNT=5000 -e USE_AUTH=true" command="exec perf:mem_over_time"

The env flag is optional, so feel free to leave that blank if you have no environment variables to set.

Updating Gems or Docker

If you change the Dockerfile or Gemfile, you'll need to run

make build

to rebuild the Docker image for your changes to apply.

Cleaning Up

If you want to reset from scratch, you can run

make clean

which will run docker-compose down, remove the bundler volume, and remove node_modules/.

Things You Probably Don't Need to Know

Infrastructure

Splits.io is built in Ruby on Rails, but has some help from other pieces of infrastructure.

                             ┌──────────────────────────────────────────────────────────┐
                             │AWS Application Load Balancer (splits.io)                 │
                             │┌────────────────────────────────────────────────────────┐│
                             ││AWS Auto Scaling Group                                  ││
                             ││┌──────────────────────────────────────────────────────┐││
                             │││AWS Target Group                                      │││
    ┌────────────┐           │││┌────────────────┐┌────────────────┐┌────────────────┐│││ Lambda tells Rails
    │AWS RDS     │           ││││AWS EC2 Instance││AWS EC2 Instance││                ││││ to parse the file
    │┌──────────┐│           ││││┌──────────────┐││┌──────────────┐││                ││││  │
    ││PostgreSQL││◀──────┐   │││││Docker        ││││Docker        │││                ││││  │  ┌──────────┐
    │└──────────┘│       │   │││││┌────────────┐││││┌────────────┐│││                ││││◀────│AWS Lambda│
    └────────────┘       ├───┼┼┼┼┤│Rails Web   ││││││Rails Web   ││││      ...       ││││     └──────────┘
 ┌───────────────┐       │   │││││└────────────┘││││└────────────┘│││                ││││           ▲
 │AWS Elasticache│       │   │││││┌────────────┐││││┌────────────┐│││                ││││           │
 │┌─────────────┐│       ├───┼┼┼┼┤│Rails Worker││││││Rails Worker││││                ││││           │ New file
 ││Redis        ││◀──────┤   │││││└────────────┘││││└────────────┘│││                ││││           │ trigger
 │└─────────────┘│       │   ││││└──────────────┘││└──────────────┘││                ││││           │
 └───────────────┘       │   │││└────────────────┘└────────────────┘└────────────────┘│││           │
         ┌───────┐       │   ││└──────────────────────────────────────────────────────┘││       ┌──────┐
         │AWS SES│◀──────┘   │└────────────────────────────────────────────────────────┘│       │AWS S3│
         └───────┘           └──────────────────────────────────────────────────────────┘       └──────┘
             │                                             ▲                                        ▲
             │                                             │ HTTPS, WebSockets                      │
             │                                             ▼                                        │
             │                                          ┌────┐                                      │
             └─────────────────────────────────────────▶│User│◀─────────────────────────────────────┘
              Sends "I forgot my password" emails       └────┘  File uploads/downloads (runs, race
                                                                attachments) via S3 presigned URLs

Not pictured:

 - beta.splits.io, an AWS Application Load Balancer with an identical hierarchy
 except pegged at 1 instance, pointing to the same external infrastructure

 - livesplit-core, a Rust library with Ruby bindings that gets deployed to
 containers so Rails Web can call it to parse run files

 - AWS CodePipeline, which calls out to AWS CodeBuild and AWS CodeDeploy to
 build and deploy code on pushes to main

Rails will synchronously parse any unparsed run before rendering it, but the asynchronous Lambda job is the preferred way for runs to be parsed because it still catches unvisited runs (e.g. in the case of a multi-file upload via drag-and-drop).

In development PostgreSQL and S3 are also Docker containers (see docker-compose.yml). Lambda is not yet implemented in development mode.

Favicons

Favicons are generated by Favicon Generator and its Rails gem. To generate favicons from the source image (public/logo-imageonly.svg), run

docker-compose run web rails generate favicon

Config for this generation is at config/favicon.json.

Theme

Splits.io runs vanilla Bootstrap 4. Historically we used one of a few themes to give it some additional distinction and a professionally tailored dark mode, but we decided to switch to vanilla with our own dark mode after several bad experiences with those themes slowly falling more and more out of date from mainline Bootstrap.

Responsible Disclosure

If you find a security vulnerability in Splits.io, please email it privately to [email protected], as posting the vulnerability in public may allow malicious people to use it before it can be fixed. We take security matters very seriously and respond quickly to disclosures.

Library Information

LiveSplit Core

Splits.io uses livesplit-core for parsing runs. The parser is located in lib/parser/*. To upgrade it, run

make update_lsc

and commit the changes.

Highcharts

To generate run history charts Splits.io uses Highcharts, which requires a written license. Licensing is based on the honor system, so you do not need to enter a key anywhere. Highcharts is free to use for testing purposes.

splits-io's People

Contributors

batedurgonnadie avatar bgreenacre avatar bigfoott avatar cryze avatar davygora avatar dependabot[bot] avatar drekdrek avatar emdantrim avatar gitkrystan avatar glacials avatar jamacanbacn avatar julioolvr avatar kenany avatar michaeljberk avatar moorecp avatar paulschnau avatar spiraster avatar wooferzfg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

splits-io's Issues

Always show run time

Currently run time only shows on mouseover of the splits timeline. It should be shown always.

Downloaded splits should be encoded into the downloader's choice of format

Currently only the original format of the file is served. Clicking "download" should give options about what format (WSplit, Time Split Tracker, SplitterZ, etc.) the file should be downloaded in.

For implementation, I'm thinking each format could be represented as a Rails view that's rendered as a file (thus giving the browser a download, not a web page) rather than as text. Something like (in HAML, for WSplit)

Title=#{@run.title}
Attempts=#{@run.attempts}
Offset=#{@run.offset}
Size=#{@run.size}
- @run.splits.each do |split|
  #{split.title},0,#{split.runtime},#{split.time}

This would make writing the format encoder basically the same process as writing the format page (which we already do).

Llanfair support

Gnintendo gave me a decompiled Llanfair and it'd be nice to use this to get support for it.

Tests

We should probably have some tests. Among them should be a test for each upload format and a test for each download format.

There's no guarantee that the downloaded files will be exactly the same file as the uploaded ones (because we lose some information, like local image paths, in some formats), but there should be a guarantee that cycling those new files through the system again (re-uploading the downloaded splits then re-downloading from each of those uploads) are exactly the same.

WSplit parsing incorrectly calculates duration

Each line of a split in WSplit reads as follows:

Name,OldTime,RunTime,BestTime

Currently, the "Duration" column is calculated by assuming the BestTime number, which can lead to incorrect durations like so. Input below:

Title=Column 10
Attempts=843
Offset=0
Size=121,25
Episode 10,0,49.26,0
Episode 11,0,133.81,0
Episode 12,0,206.37,0
Episode 13,0,283.61,0
Episode 14,0,393.22,0
Episode 15,0,463.3,0
Episode 16,0,544.67,0
Episode 17,0,665.22,0
Episode 18,0,775.6,0
Episode 19,0,877.24,0

(in fact, even the actual splits show incorrect durations as well, since not all segment times equal the difference in split times - they represent best splits from other runs)

Offer to disown splits if you beat a PB

The LiveSplit format holds game and category fields, so if a user uploads splits with the same game and category as one of their existing runs, but a better time, we can offer to disown the old splits so they don't clutter up the user's run list.

Prevent `NaN%` widths for blank runs

Other than for testing I'm not sure why you'd upload a blank run, but if you do we end up dividing by 0 all over the place which means we get bars with NaN% widths.

I guess the most sane way to display the splits timeline would be to give each split an equal amount of space by using widths of 100/num_splits%. Obviously this isn't technically accurate, but the only accurate way to handle this would be to blow up. So it'll have to do.

Fix Firefox

The timeline is all messed up in Firefox.

Remove local icon paths from uploaded splits

Some splitting programs allow icons to be displayed next to splits. To do this, the program saves the local path to the image in the splits file, which we end up parsing. However the original splits file remains intact on our end, and if someone chooses to download it, they see those local image paths.

This is one part minor privacy concern and one part minor feature addition.

Minor privacy concern

Usually people won't be revealing any private information in just a path, but you never know. And even if they aren't, it's not something we should really be storing. We don't need it, we shouldn't be using it, and it's generally just not something that should be sitting around not doing anything.

Minor feature addition

Everyone who downloads the run (other than the original runner) will have broken image paths. It's up to the splitting program how to handle a broken path (it may complain, it may display a "broken image" image where the image should be, or it may just pretend like you never specified a path), but it'd be better if we weren't supplying it with one in the first place.

Going about it

When #3 gets implemented, we will have a splits encoder which can convert parsed splits back to any file format. When encoding, instead of placing the icon paths that we have, the splits encoder should fill them in indiscriminately as blank paths.

When that's in place, we will be able to remove icon paths by simply decoding the uploaded splits file then re-encoding it back to the same format. This might not be the most efficient thing in the world, but it will be a simple way to make sure that the splits we store match up with the ones we hand out to downloaders, which may help prevent other issues later on down the line. You know, in an eating-your-own-dog-food sort of way.

Don't list duration for splits after missed splits

When you miss e.g. an 8-minute split and it is immediately followed by e.g. a 10-minute split, the 10-minute split's duration is displayed as 18 minutes. Obviously this is because since you missed the split, we have no idea where the first stopped and the second started. But displaying 18 minutes as the duration for the second split is incorrect.

We should either not display this duration, or display it and signify somehow that it was the pair that took 18 minutes, not just the second split. If we do the latter, this becomes slightly more complicated if two splits in a row are missed.

Add a fallback upload form

As a fallback for those without JavaScript or without file managers, and also as a backup for client or server issues, there should be a fallback upload form that doesn't use JS and that can be access from a link on the upload page.

It should be located at /upload/fallback.

LiveSplit downloads are broken

Downloadable splits don't load correctly into LiveSplit. Not sure why yet, but I'm suspecting it's because the Time.at calls are adding a space to the fronts of times.

Timeline segments should expand or show tooltips on mouseover

Some splits are so small relative to the run that the width allocated to them in the timeline chart is not nearly enough to display the split title or duration.

Example: http://splits.io/YtJF

Simple solution

Add a mouseover tooltip to each split, and display that split's title and duration in the tooltip.

Pretty solution

Make splits smoothly expand their width on mouseover to be wide enough to display their information. The widths of all other splits would need to temporarily shrink to compensate, which in itself could be a cool animation.

Alternative pretty solution

The split could expand on top of the other splits and gain a box-shadow to make it look as if it is above the others. An animated box-shadow that increases from 0 over a second or so could make it look like the split is being raised out of the timeline bar. This solution would be pretty, but might be too flashy for something as simple as a mouseover.

Fill out an API

There should be a JSON API for getting information from splits.

Logging in shouldn't change your page

Right now logging in always takes you to the front page. We should save the page you're on in a cookie or session or something so when Twitch sends you back to us after authentication, we know what page to load.

Timeline widths can be assigned "NaN%"

If a WSplit run's final split has a finish time of 0 (i.e. the run was never completed) all segments of the timeline chart get a NaN% width.

This is because we obtain the total run time by looking at the final split's finish time. To get each timeline segment's width, we divide that split's duration by the total run time, which gives us a divide-by-zero error in this case.

This only happens for WSplit runs because SplitterZ and TST don't keep track of finish times, only durations, so for those we calculate them manually by summing individual split times.

Example: http://splits.io/F9X3

Pretty colors, but not quite accurate.

The simple fix is to calculate WSplit run times like we do for SplitterZ and TST, instead of depending on that last split's finish time.

However this also brings up an issue of whether or not to display the timeline chart as we currently are for incomplete runs, as its total width doesn't actually represent the duration of the game -- only the total duration of the completed runs. This might be something to think about later.

Downloaded splits should have their format as an extension

Currently there's no way for the user to know what program the splits they're downloading (by accessing /:nick/download) came from. Maybe eventually we will convert between formats, but for now it should be apparent what program they can be loaded into.

The downloaded file is currently named :nick (e.g. 'pmFs'). It should instead be named :nick.:format (e.g. 'pmFs.wsplit').

Uploading from LiveSplit doesn't bind splits to your account

When you upload from within LiveSplit, the splits won't bind to your account if you're logged in. The fix for this goes hand-in-hand with the fix for #29 -- if the run has 0 hits and is unclaimed when it receives its first hit, we will auto-bind it to the account that visited it.

The slight difference between this solution and the solution to #29 is that the latter applies when the first hit is anonymous. The former applies when the first hit is logged in.

Comparisons

Allow comparisons to other runs. These should involve seeing the timeline for the compared run immediately below your run's timeline, as well as splitting program style "+XX:YY" and "-XX:YY" notes near each split time and duration.

URLs could look something like /:run/compare/:another_run, /:run/:another_run, or /compare/:run/:another_run. It could be nice to compare > 2 runs at some point in the future -- although I'm not planning on this quite yet, it should be kept in mind when choosing a URL format.

Potentially significant runs to compare against are:

  • The best (on file, for that game/category)
  • The next-best run to yours (your run's "better neighbor")
  • Your previous run
  • Your PB

Some or all of these should be built-in options displayed on the run page. Comparing to any other run can just be a manual process of manipulating the URL.

Nearly necessary changes for this to happen: add game, category, and time fields for runs to the database.

"Useless stats"

Maybe:

  • Mean split
  • Median split
  • Box plot maybe? Is this overkill?
  • Shortest split
  • Attempt #
  • Exact finish time (millisecond granularity)
  • Number of splits
  • Total life playtime

Clean up CSS

The CSS is getting a bit too messy and centralized and should be organized and split up into different files.

Allow an anonymous run to be claimed if it was just uploaded

If you're logged out and you upload splits, you should be able to log in and claim them. You should only be able to do this if those splits have exactly 1 hit (your current page load) to prevent false claims. So, you'd have to do it right away.

A clean solution to #70 will solve this.

Random can direct to the current run's splits

While uncommon (increasingly so when increasing numbers of runs are added), navigating to another random split (via the random button, for example) can send you back to the one you're currently viewing.

Clean up all these `Time.at` calls

Basically everywhere that uses human-friendly time formats (e.g. HH:MM:SS.NN), which includes the show view and some of the downloadable formats, needs to call something resembling Time.at(some_duration_in_seconds).utc.strftime("%k:%M:%S.%N").strip, which makes for a lot of (sometimes minorly different) identical calls.

We should abstract this to something that we can do things like split.duration.minutes or split.duration.seconds on, or maybe even split.duration.pretty if it turns out that all of the formats expect the same style string.

Add a faq

Add a short faq. The feedback and GitHub links in the footer should be rolled into the faq as questions and removed from the footer. Also could add a link to some tracking stuff that I've been adding!

Uploaded file size should be checked before we do anything

If someone happens to upload a gigabytes-large file we shouldn't try to parse it. We should ditch the file right away and complain at the user, because they deserve to be complained at for that.

You heard me, users. Who would do that.

Anything over 1 MiB is probably suspect.

Views for different formats should be combined into one

Setting up different views for each format was planned so that splits file formats could be extremely different without having to muck up a single view with a bunch of conditionals.

But now that three major formats are implemented with minimal differences (SplitterZ and TST can, right now, work off of identical views, and after #6 is solved, WSplit will be able to as well), it's becoming abundantly apparent that having a view for each format is overkill.

They should be combined into one view for displaying splits from any format, and it should be called show.html.haml.

Accounts

Implement user accounts! If you're logged in when you upload splits, those splits will be tied to your account so that you can find/delete them later.

WSplit parser should accept OldTime values

Right now the WSplit parser expects a 0 as the second value for each split:

Captain Jelly,0,1256.61,719.44

But it can be a nonzero value if the user uses WSplit's "Set this run as old" function. We should accept a normal-looking time there. We won't be using it, but the parser should consume it instead of panicking.

Recognize when identical splits are being uploaded

If a file being uploaded is identical to one that's already been uploaded, we should drop the new file and direct the user to the existing splits.

To do this we could hash each file when it's uploaded and store the hash in the database. Then whenever new splits are uploaded, we check the database for their hash.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.