juliapackaging / pkgserverloganalysis.jl Goto Github PK

Once we nail down the data schema in #1, we'll need to host and serve the data from somewhere. If we end up with a SQL-like interface, a public MariaDB interface might be the simplest way, with a few protocol adapters slapped on front so that we can serve an api over HTTPS that gets the TLS stripped off, the HTTP request is parsed, and a SQL request can be made.

Let's decide on a database server to serve from and actually host it somewhere. (MIT OpenStack probably?)

It would be great to have a new release of the CSV with downloads per package. @jkrumbiegel and @Datseris expressed interest on Slack, and I and I’m sure others would be interested too.

We need to come up with a data schema for passes to dump into a database.

Right now, I'm thinking each data analysis gets its own database, with its own schema. Each analysis pass will be responsible for outputting a CSV-like output, (e.g. "downloads per day, per UUID, per julia version") and a database engine (such as MySQL) will serve queries upon those databases to the world. We'll run the analysis once per day on the last month of data, and as we add new analyses/fix bugs, we'll update that entire time window with the better results, while historical results will just be frozen in time.

We should have a CI job that runs the analysis passes on the anonymized data testset, to ensure that it can run without erroring.