http://angular-ui.github.io/bootstrap

write tcp 127.0.0.1:5000->127.0.0.1:43217: write: broken pipe

This doesn't matter, along with a few others. So we should just filter these out from panicing.

Expose data downloaded from githubarchive.org

I had difficulty determining what a "contribution" was beginning this. I definitely got all the ones that I considered contributions, but may have included too many. For example I think commenting on an issue counts as a contribution, but other people don't. So make a selection list where people can define which event types they consider to be "contributions" before querying.

I have over 1000 unread emails from processing files on githubarchive.

Currently when invoked, githubarchive-util will try to download every file from 2015-01-01 to present. There are a few that won't work (see igrigorik/gharchive.org#101). Error messages should not be generated for these.

For the past 2 weeks, the processor has been down because Arch changed the Python version and I didn't realize it, so I had to catch up processing a million or so records. I need the github-contributions-process to write the time of the last record processed to a www-accessible file, then have a service that polls for that every hour and makes sure that the output is reasonable.

The way I monitor the server now is a mess. Recently the processor broke because a file was missing from githubarchive.org. Normally the server emails me every hour telling me the results of the archive-processor run, and separately it emails me if it realizes that the latest event is more than a couple hours old, but at some point it stopped doing that because I chmod'd the ssmtp.conf wrong, and the logs on the disk aren't something I look at regularly until I realize something's wrong.

So I need to look into alternatives and implement one of them. Requirements:

• Free (at least for this usage)
• Easy to post arbitrary events through an API
• Python and Go libraries (preferably drop-in replacements for the standard logger)
• Must have a command line utility to post these events, preferably one with no dependencies

Because we decided not to use Digital Ocean connecting to another VPS over a WAN, couple of minor issues:

• Revert change that removed raw archive data from NGINX
• Revert change that removed raw archive data links on index.html
• Remove DO branding for the time being

Make script to provision to Ubuntu 14.04

Since I was already warned that the site was slow, I didn't mind that I was still waiting for results after a minute. But as the site was still showing its cubic spinner after ten minutes, I looked in the Dev Tools that showed a 500 Error ...

So the query probably failed quite early, but I was not warned. Handle the error and tell the user you are overloaded (or whatever).

There's tons of unnecessary complexity in the Flask stuff, plus I want to get better at Go.

We need MongoDB to work over a WAN so that I can host the DB on Vultr and still have DO sponsor the frontend.

I don't know what I was thinking when I used Arch as the server. That was a bad idea and it's broken several times. Use the ansible script to provision a new box.

DB: MongoDB , /github-archive stuff, archive-processor
Web: Everything else

This is because of DigitalOcean's offer to sponsor the project, but I can't fit all the data on their droplets.

Splitting the DB away from the web layer means that I can't link to the raw data for now without setting up a web server on the DB. So temporarily remove this until Digital Ocean has storage instances (https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/3127077-extra-diskspace)

Right now we serve some out of other CDNs, and the rest are individual, non-minified versions served locally. No attention was paid to optimizing anything.

Look into webpack

This just occured to me. The pre-2015 events (in the timeline directory) don't have event_id attributes. However, the new ones all do. Maybe I could replace the MongoDB _id attribute with event_id for the post-2015 events. Dropping that index would likely result in a huge increase in insert performance, which we really need. Right now there are 4 indexes on that collection, and not being able to fit them in memory is what really slows things to a crawl.

Thoughts, @joshjordan ?

Right now you can't link to results. This is terrible.

If the processor crashes, an incomplete gzip is left in the output directory. That stops the rest of the archive from being processed on the next invocation, because if the file is in the destination directory and not empty, it thinks it's done.

The output archive should be written to a temporary file, then moved atomically to the destination.

This looks terrible on mobile devices. Need to fix:

• Make header and hamburger fit inline
• Move search out of hamburger menu
• Make sure that top header part isn't too big (should be fixed by first item) such that it covers page content
• Get footer down to one line
• Kill Fork me banner on mobile

Time series data would be cool to display here since it looks like lots more events are occurring recently.

When the server is down, the responseError handler is still called. But then rejection.data is null, which causes another error! 😢

    function ConfigureErrorHandler($httpProvider) {
        $httpProvider.interceptors.push(function($q, $rootScope, $log, $injector) {
             return {
                 'responseError': function(rejection) {
                     $log.debug(rejection);
                     $rootScope.errorDescription = rejection.data.error;

It has all the droppings from uwsgi and stuff on it. Gross.

Steps:

• Search for someone
• Look at their events
• Search for someone else
• Some of the columns (e.g., repository) are still from the first person. Mysterious!

When loading the page, all the angular template code briefly flashes. Use ngCloak to stop that.

Google doesn't really care about this site, in part because it can't see it has any content. There's no list of users anywhere (since it's based around search), so it won't crawl users' pages. Fix that with a bunch of sitemap.txt files.

The IP addresses belong logged are all localhost, because nginx is proxying everything. This is stupid; I need to fix it.

Make a deployment script!

Clean up this mess I've made of Angular! Get rid of explicit $http.

@igrigorik needs some files with duplicate events de-duplicated. Make a utility to do this.

Since queries are only (currently) performed on the _user_lower key, try creating static files for each user. So the query "hut8" would end up opening up "hut8.json.gz". Yes this is insane but maybe it would actually work better than MongoDB.

Part of #24 involves simply counting events for each hour, which is a pretty decent-sized task, so it should be its own issue. It would be pretty crazy to just use aggregates over the entire collection.

Blocks #24

Use letsencrypt!

Right now the "repo" page just displays a list of repositories that the user contributed to without any indication of how many contributions were made to each. Fix that by making it a table, sortable by either column.

Creating branch feature/angular for this.

Compared to the enormous amount of data you are working with I would assume that the (presumably) little traffic that's hitting you would be manageable to cache for each user (to some extent).

For interested developers that would poll your site regularly, this should lower the load quite a bit.

From nginx error log:
712#0: *6020 upstream prematurely closed connection while reading response header from upstream ...

This seems to happen after some amount of time. I think I'm handling mgo sessions wrong and eventually it will just stop working.
Also, the logs aren't adequate at all; there are no errors whatsoever in there.

The /stats endpoint is unused, and it would be nice to be able to show some statistics about the whole collection on the page. So add a sticky footer and put it in there.

Right now there are no indexes on the new server. Definitely needs to be fixed.

The provision script should run this script (asynchronously?)

An outage at GitHub causes githubarchive's .json.gz files not to be where this expects them (since it returns 404s), so currently the archive processor just errors out constantly.

Whoops, obvious bug with events cache:

Search for one user
Visit events page 2.
Search for another user
Go to events tab
It displays original user's event page .

I think builds should have no external dependencies, and the built-in Go toolset introduces tons. See this for details:

https://getgb.io/

I think it's the best go build tool.

Because there are 8,760 hours in a year, there are starting to be quite a few files in a single directory. Several parts of the processor iterate over everything and this is causing performance issues.

Right now the dates exported by the archive processor are stored as strings in ISO8601. They should use this syntax:

{ "created_at": {"$date": "2012-01-01T15:00:00.000Z"} }

instead of:

{ "created_at": "2012-01-01T15:00:00.000Z"}

Ideally deployments should work like this:

• Somehow get a binary in the right directory on the server (/home/ghc/ghc-app/bin/)
• It should be named ghc-app-<sha1> where the sha1 is the first few hex-encoded bytes
• Binding should always be done with SO_REUSEADDR
• Side idea: bind each build to its own port that is derived from the SHA1 of the binary (Python 3):

app_bytes = open(r'path/to/ghc-app', 'rb').read()
app_hash = hashlib.sha1(app_bytes).digest()
port = int.from_bytes(app_hash, 'big') % (65535 - 1024)

• Start the binary as a child process of the deployment thing listening on a test port on localhost (9001 maybe, since it's over 9000)
• Run some basic tests on that to make sure it returns /stats for example
• Kill the test process
• Switch the link to ghc-app to the new build
• Make a note of the old ghc-app's PID
• Start the new ghc-app just like the old one, again using SO_REUSEADDR. So then two processes listen on the same port.
• Kill old ghc-app