pypi / linehaul Goto Github PK
View Code? Open in Web Editor NEWARCHIVED, replaced by https://github.com/pypa/linehaul-cloud-function/
Home Page: https://github.com/pypa/linehaul-cloud-function/
License: Apache License 2.0
ARCHIVED, replaced by https://github.com/pypa/linehaul-cloud-function/
Home Page: https://github.com/pypa/linehaul-cloud-function/
License: Apache License 2.0
In linehaul, there are 3 states that any particular event can be in:
For (1) the correct outcome is obvious, we have data so we want to save it in BigQuery.
For (2) the current thing we do is record a download, but with all of the data that typically comes from the user agent missing. Thus the BigQuery table more accurately reflects all of the downloads, but projects querying the data needs to be more careful about how it queries the data (it's easy to do something like py3_downloads / total_downloads
, however that would incorrectly give a smaller percentage, since it would count unknown as py2). Prior to Linehaul v3, the behavior was to throw away this event and not log anything for it.
For (3) Linehaul v3 and previous throw away the event (we implement this as "ignored" user agents). The list of these can be found at:
So ultimately the question is, do these behaviors make sense? Which boils down to whether we want BigQuery to most accurately reflect every download, or whether we want to filter the data to data that is more usable for specific questions (but of course, less usable for other questions that are likely to be more of an edge case).
Thoughts?
I ran a script today from 01:07 to 09:39 UTC, downloading a file every six minutes. It made a total of 86 downloads, but only 40 of them are in the database:
SELECT *
FROM
TABLE_DATE_RANGE(
[the-psf:pypi.downloads],
TIMESTAMP("2018-05-25"),
TIMESTAMP("2018-05-25")
)
where file.project = "six" and file.version = "0.9.0"
and country_code = "DE" and details.installer.version = "10.0.1"
order by timestamp desc
Looking at the data more generally, there are 1440 minutes in a day, but on 05-23 only 748 of them had any data recorded:
#standardSQL
SELECT timestamp_trunc(timestamp, MINUTE) minute, count(*)
FROM `the-psf.pypi.downloads*`
where _table_suffix between "20180523" and "20180523"
group by minute order by minute desc
Browsing through those results shows many extended gaps in the data, e.g. from 22:45 to 22:54, and 21:36 to 22:16.
I know this service isn't widely-known, but isn't it intended to be a complete record of all PyPI downloads? If so, something is clearly going wrong.
As mentioned by @ncoghlan in pypa/pip#5424 (comment):
submit a PR to update
linehaul
'sUserAgent
schema to include an extra optionalwrapper = pyrsistent.field(type=Installer, factory=Installer.create)
field
The motivation is to allow wrappers, like pipenv, to measure adoption of their tools.
The project has since switched to attrs, but the idea is the same. The change should be somewhere around here.
i'd like to use this library!
Hi!
It might be interesting to be able to query through failed installation data, if such a thing is possible.
I.e. -- to view pip install foo
by volume, for foo
a non-existing package, or existing package with no uploaded files.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.