Comments (13)
Can you post some of your source code so we can see what you're trying to do? If you're using the client (by Andy Casey), it's possible you're accessing other end points that have separate rate limits, and may be giving you this response.
from adsabs-dev-api.
Yes, I was running his beers-for-cites.py script
from adsabs-dev-api.
I had similar symptoms when running a script that uses Andy Casey's ads python module to issue repeated queries for DOI or arXiv identifiers and bibtex even though (below) I'm not close to the limit. The script I'm running is [https://github.com/dwillcox/pybib/blob/master/pybib.py]
Simplified Code
def query_ads(self, query):
paper_query = ads.SearchQuery(**query)
paper_list = []
for p in paper_query:
paper_list.append(p)
self.paper = paper_list[0]
self.bibtex = self.paper.bibtex
self.query_ads({'identifier':'arXiv:1507.01927'})
Traceback
Traceback (most recent call last):
File "/home/eugene/codes/astro/pybib/pybib.py", line 396, in <module>
dc = DocumentCollection(args.infiles)
File "/home/eugene/codes/astro/pybib/pybib.py", line 311, in __init__
self.documents = [Document(f) for f in files]
File "/home/eugene/codes/astro/pybib/pybib.py", line 197, in __init__
self.query_ads({'identifier':self.arxiv})
File "/home/eugene/codes/astro/pybib/pybib.py", line 267, in query_ads
self.bibtex = self.paper.bibtex
File "/home/eugene/local/anaconda2/lib/python2.7/site-packages/werkzeug/utils.py", line 73, in __get__
value = self.func(obj)
File "/home/eugene/local/anaconda2/lib/python2.7/site-packages/ads/search.py", line 231, in bibtex
return ExportQuery(bibcodes=self.bibcode, format="bibtex").execute()
File "/home/eugene/local/anaconda2/lib/python2.7/site-packages/ads/export.py", line 62, in execute
self.session.post(url, data=self.json_payload)
File "/home/eugene/local/anaconda2/lib/python2.7/site-packages/ads/base.py", line 39, in load_http_response
raise APIResponseError(HTTPResponse.text)
ads.exceptions.APIResponseError: u'Rate limit was exceeded'
Rate Limits after Traceback
< X-RateLimit-Limit: 5000
< X-RateLimit-Remaining: 4896
< X-RateLimit-Reset: 1458777600
from adsabs-dev-api.
Ah, okay, so I looked at the ads code a little more and since I get this error when trying to get bibtex, I see the ads module is accessing https://api.adsabs.harvard.edu/v1/export/bibtex
Doing:
curl -v -H "Authorization: Bearer [my token]" 'https://api.adsabs.harvard.edu/v1/export/bibtex'
Returns:
[...]
HTTP/1.1 429 TOO MANY REQUESTS
[...]
< X-RateLimit-Limit: 100
< X-RateLimit-Remaining: 0
< X-RateLimit-Reset: 1459036800
[...]
Thanks, @jonnybazookatone is correct in my case, I can make only 100 daily requests for bibtex. Once I generated a bibliography for my existing collection of PDFs, it's not so bad, since I'm unlikely to add 100 articles/day to my collection as of yet!
I don't know which endpoints are called by the beers-for-cites script, but the API urls you can try with the curl -v -H etc command are here: [https://github.com/andycasey/ads/blob/4871fc6c272f872a8324e24881b88ac4511ea544/ads/config.py]
from adsabs-dev-api.
Could the 100 be increased to 1000 or 400? I tried to programmatically build a bibtex collection of all papers that cite me, and ran into that problem.
from adsabs-dev-api.
The bibtex endpoint receives a list of identifiers, so you should send batch of identifiers (1 request; instead of N requests with 1 bibcode each). If the beers-for-cites script has that behaviour, I'd suggest making a PR to address that problem there.
from adsabs-dev-api.
Yes, @dwillcox, you're hitting your API limit on the /export
end point when you're using ExportQuery().execute(). You're better off, as @romanchyla points out, first collecting your bibcodes and then sending them to export query. For example,
q = ads.SearchQuery(q='star', fl=['id', 'bibcode'])
bibcodes = [article.bibcode for article in q]
bibtex_query = ads.ExportQuery(bibcodes=bibcodes, format='bibtex').execute()
This requires 1 request, and so will only reduce your rate limit on /export
by 1. If you do it for each bibcode separately, you'll obviously reach the limit at 100 bibcodes.
from adsabs-dev-api.
@JohannesBuchner, so you're best off trying to collect the bibcodes you need, and then send them to the export service as I've shown in the response to @dwillcox. If you were simply wanting to find all the papers that cite your papers, you can do this using the citations
operator. For example,
q = ads.SearchQuery(q='citations(first_author:"Buchner, J." database:astronomy)', fl=['id', 'bibcode'])
This will return all the bibcodes that cite papers with first author "Buchner, J." in our Astronomy database, using 1 request, and 1 more if you want to export them . However, this will run into author disambiguation, as it looks like there are a lot more J Buchners around. If you're only interested in yourself, and who cites you, you could also think about using ORCiD to identify papers that are yours. You would then simply do:
q = ads.SearchQuery(q='citations(orcid:0000-0001-8043-4965)', fl=['id', 'bibcode'])
and then send them to export. You can read about ORCiD here: http://adsabs.github.io/help/orcid/claiming-papers
Do note I've only requested 'id' and 'bibcode', if you start accessing other contents via the attributes, ie., Article.title
this will send another request, and use up your rate limit. To avoid this, you can request more field parameters on your initial request, fl=['id', 'bibcode', 'title']
and so on.
Finally, I would say this issue is mostly related to these two: andycasey/ads#43, and andycasey/ads#44. So keep an eye on here for changes that may help you avoid hitting rate limits due to unoptimised code.
from adsabs-dev-api.
Thanks so much for the quick replies and clear explanations, @romanchyla and @jonnybazookatone!
I didn't realize I could pass a list of bibcodes into ads.ExportQuery(), that's very helpful and I will change my script to do so.
My script calls pdfgrep to get DOI or arXiv ID's from PDFs and then builds a bibtex bibliography for me, and your API saves a lot of time. Thanks for the great work!
from adsabs-dev-api.
That sounds like a reasonable solution. To try it out, I ran in Bumblebee
citations(orcid:"0000-0003-0426-6634" AND first_author:"Buchner, J.")
Unfortunately, this returns many papers that are somehow related but actually do not cite me if you click into them ...
from adsabs-dev-api.
Can you give some examples? I looked at the first 10 documents returned and they all have citations to you in their References section.
Also, a quick glance at the numbers seems to make sense. When you search without citations()
you see you have two papers with 35 and 37, and with citations()
it returns 65 documents. The reduced number is papers that cite both of them, for example: 2015MNRAS.453.1946G
.
If you want to also know you total citation_count
(ie., 35+37=72), then you can send your ORCiD bibcode list to the metrics end point:
q = ads.SearchQuery(q='orcid:0000-0003-0426-6634 first_author:"Buchner, J."', fl=['id', 'bibcode'])
bibcodes = [article.bibcode for article in q]
bibtex = ads.ExportQuery(bibcodes=bibcodes, export='bibtex').execute()
metrics = ads.MetricsQuery(bibcodes=bibcodes).execute()
print metrics['citation stats']['total number of citations']
>>> 72
This is 3 requests.
from adsabs-dev-api.
Or, if you want to know which papers cite which document, then you can also request the citation list on your first query:
q = ads.SearchQuery(q='orcid:0000-0003-0426-6634 first_author:"Buchner, J."', fl=['id', 'bibcode', 'citation'])
articles = [article for article in q]
print articles[0].citation
>>> [u'2014A&A...571A..34V',
>>> ....
>>> u'2016PhRvL.116e1103L']
Then you can loop through them and count them. It really depends on how you're wanting to use it and change it in the future.
from adsabs-dev-api.
I'll close this ticket as the rate limits are behaving as expected. If you have any more questions on the topic, feel free to ask them here.
from adsabs-dev-api.
Related Issues (20)
- Get list of corresponding bibstems and journal full names through API?
- curl working but not request HOT 3
- Export API example custom format returns AASTeX format
- curl -H 'Authorization: Bearer:MZUyqw3ZR0aKFApe9ZWdYXB89HJcUBNvLir8751D' 'https://api.adsabs.harvard.edu/v1/search/query?q=star&fl=bibcode'
- pdf downloads with curl fail with captcha message HOT 3
- Dumping all personal libraries into a bibtex file HOT 1
- Syntax error while parsing DOI HOT 4
- Can't use != to check if an article exists in a container HOT 1
- export wth wrong xsd location and xml namespace in refabsxml format HOT 1
- What are the constraints on library names HOT 1
- Can't remove from library some papers submitted with arXiv bibcodes
- Can't get ratelimits HOT 1
- Example curl request in README does not produce X-RateLimit headers HOT 4
- Match regular expression on "body" field
- Recent update? HOT 6
- Question: Is there a way to perform a query with first affiliation? HOT 1
- Big query search does not work with some bibcodes HOT 1
- Issues with CSL export HOT 2
- Add reference_count as a fl field in searches?
- Citation field is not filtered by citation count HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adsabs-dev-api.