adsabs / adscitationcapture Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU Affero General Public License v3.0
License: GNU Affero General Public License v3.0
Pubdate/year both are filled using the same source:
For instance, 2016zndo.....55143A
in solr has year":"2016"
and "pubdate":"2016-06-21"
. But date
is filled using the current timestamp:
This makes that a query like year:2016 NOT pubdate:[2016-01 TO 2016-12]
returns results, because solr is using date
and not pubdate
fields when the user queries using pubdate:
.
Implement unit tests for forwarded messages and consider expanding other tests too.
Unit tests do not currently cover the file import portion of CitationCapture. It would be good to add some tests for both DeltaComputation
and ReaderImport
when it is merged into the mainline branch so that way we can confirm the behavior of the two imports are consistent.
We need to implement a function to guess first and last names with different possible formats (e.g., "Firstname Lastname", "Lastname, Firstname") and multiple last names or middle names. Then this should be applied to address this:
Use a random TOKEN for the asclepias webhook prototype and update eb-deploy PR accordingly.
ADS Classic can output a list of citation coming from the same citing bibcode but different source. Sometimes, for one source the resolver did not work so it provides and empty cited bibcode. We must ensure that when we discard duplicates, we keep those that are resolved.
In order to remove duplicates and invalid citing bibcodes from a list of citations, two options are available:
If we go with option 1, we would use the mapping between all known bibcodes and canonical bibcodes, which is found in file /proj/ads/abstracts/config/bibcodes.list.all2can
The first column contains all the known bibcodes in ADS, and the second column contains their canonical version. Given a set of citing bibcodes, one would look each one up in this list. If an entry exist, then replace the bibcode with its canonical version. If an entry does not exist, the bibcode is invalid and should be ignored. At the end of the process, the resulting list of bibcodes is uniqued.
If we go with option 2, we would use an API call to bigquery and submit our list of citing bibcodes at once. The search engine will resolve each bibcode in its canonical form and return a list of unique records. The bibcodes of these records are the ones to use as our citation list.
Since bibcodes can be remapped at any point, one would periodically need to perform this operation and update the relevant records on a regular basis (at least weekly). ADS classic does this via updates to its nonbib pipeline, so a similar model could be used by the citation capture pipeline.
{"asctime": "2019-01-07T22:01:45.260Z",
"levelname": "ERROR",
"processName": "ForkPoolWorker-3",
"message": "Failed parsing",
"exc_info": "Traceback (most recent call last):\n
File \"/app/ADSCitationCapture/doi.py\", line 208, in _parse_metadata_zenodo_doi\n
parsed_metadata = dc.parse(raw_metadata)\n File \"/usr/local/lib/python2.7/dist-packages/pyingest/parsers/datacite.py\", line 168, in parse\n
aaffils.append(aff.strip())\n
AttributeError: 'NoneType' object has no attribute 'strip'",
"timestamp": "2019-01-07T22:01:45.260Z",
"hostname": "adsvm05"}
Store results in a database as citation changes are processed.
Software source/concept doi records do not have metadata independent of the most recent version, even though they have a unique doi. As the metadata updates, We keep the bibcode year in these records frozen. This ultimately results in the bibcode year and publication date being out of sync, sometimes by several years. To address this:
maintenance_metadata
to retain the publication year for concept doiscitation_target_version
table and updates the current metadata accordingly.Direct inges to master pipeline without going through ADS Classic.
Dump a sample of ADS events in JSON format
The last name initial used to build the bibcode does not use the simple normalized code implemented here:
Remove ORCID data from affiliations and store it in a useful place.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.