Comments (7)
Thank you! I'll send you a little gift. Love your library!
from camply.
Thanks for reporting @rboarman. I'll take a look into this - we're currently rotating between a few User-Agent headers:
camply/camply/config/api_config.py
Lines 18 to 31 in 62d1a44
Hopefully this is something we'll be able to get around.
from camply.
Ahh, I have more info! This is a problem with the RIDB Database API.
❯ export LOG_LEVEL=debug
❯ camply campsites \
--rec-area 2725 \
--start-date 2022-07-12 \
--end-date 2022-07-13 \
--continuous \
--notifications pushover \
--notify-first-try
2022-01-13 10:47:50,836 [ CAMPLY]: camply, the campsite finder ⛺️
2022-01-13 10:47:50,842 [ INFO]: 1 booking nights selected for search, ranging from 2022-07-12 to 2022-07-12
2022-01-13 10:47:50,844 [ INFO]: Retrieving Facility Information for Recreation Area ID: `2725`.
2022-01-13 10:47:50,852 [ DEBUG]: Starting new HTTPS connection (1): ridb.recreation.gov:443
2022-01-13 10:47:50,908 [ DEBUG]: https://ridb.recreation.gov:443 "GET /api/v1/recareas/2725/facilities?full=true&offset=0 HTTP/1.1" 403 919
2022-01-13 10:47:50,910 [ INFO]: {'accept': 'application/json', 'apikey': 'XXXXXXXXXXXXXXXXXXX'}
2022-01-13 10:47:50,910 [ ERROR]: Receiving bad data from Recreation.gov API: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: uWqHopzPrg1JtSkQPWOfThlGmwJIVPqmmAfRNN9oZMniNvv9v-rxrg==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
camply
uses two APIs to talk to recreation.gov:
- https://ridb.recreation.gov
- This is the metadata Database / REST API for Recreation.gov - we get campsite, campground, and recreation area metadata from this API
camply
uses an API Key embedded into the software to get this data. You can override this with yourRIDB_API_KEY
environment variable
- https://recreation.gov/api/camps/availability/campground/
- This is the undocumented Campsite Availability API that
camply
uses - There is no authorization on this API - this is where we randomly select a
User-Agent
header to make requests
- This is the undocumented Campsite Availability API that
The strangest thing about this issue is the RIDB Database API Landing Page is displaying the following warning:
Due to high volume of API calls, cache time for RIBD’s camping-availability endpoint will increase from 15 seconds to 1 minute moving forward. Please reach out to the Contact us center for any issues in regards to API disconnecton or downtime.
My understanding though is that the camping-availability endpoint doesn't live on the RIDB 🤷♂️. I'll keep digging and see what I can find and follow up here
from camply.
Thank you. My calls were hitting every five minutes which should be fine right?
One thing to consider, is to allow us to pass in an array of user agents via the .camply file. That might help.
Thank you for the assistance!!
from camply.
@rboarman you were absolutely right. The RIDB API is kicking back the User-Agent passed from the Python Requests Package - they haven't documented this anywhere but that looks to be the problem. It's probably just a matter of time until it invalidates the attached RIDB API Key as well -- I'll look into having it choose a random API key from a few options soon.
This PR (#45) adds a ton of User-Agents between Chrome, FireFox, and Safari and implements the RIDB API to randomly choose one for each API call.
from camply.
I closed this out - tests are passing on my end and the fix has ben released to PyPi and Docker. A simple pip install --upgrade camply
should fix everything on your end (once this workflow has finished). Please let me know or re-open this issue if you have any problems. Thanks for reporting @rboarman
from camply.
Thanks - so happy you like it. I've tried to build camply to be respectful of the APIs and services it uses - this includes rate limiting and exponential backoff. However this is the first I've seen of recreation.gov pushing back and trying to crack down on tools like camply hitting their API - I wouldn't be surprised to see more bugs like this coming out in the future.
from camply.
Related Issues (20)
- "No campgrounds found to search" when using valid campground # HOT 4
- ReserveAmerica.com HOT 3
- Provider Request: add support for `freecampsites.net` HOT 1
- SearchAlabamaStateParks returns 404 HOT 1
- Is there a way to track event ticket? HOT 2
- GoingToCamp fails for WA HOT 4
- ReserveCalifornia cancellations are locked HOT 1
- Allow searching across multiple providers by allowing multiple YAML files in same query (Enhancement Suggestion)
- ReserveCalifornia does not detect available campsites HOT 3
- Sonoma County, CA Regional Parks Addition HOT 2
- RecreatDotGov campsites search returns false positives HOT 2
- Scanning Yellowstone crashes with "AttributeError: 'list' object has no attribute 'items' HOT 5
- Yellowstone Availability before start date? HOT 1
- Pushover Out of Credits HOT 4
- Start and End Date Windows do not behave as expected HOT 1
- Continuous Search not working with YAML file HOT 4
- why is push disabled on your dockerbuild? HOT 4
- ReadTimeout: HTTPSConnectionPool(host='camping.bcparks.ca', port=443): Read timed out. (read timeout=30) HOT 4
- ConnectionError: HTTPSConnectionPool(host='camping.bcparks.ca', port=443): Max retries exceeded with url: /api/availability/map?mapId=-2147483647&resourceLocationId=-2147483647&bookingCategoryId=0&startDate=2024-07-27&endDate=2024-09-02& isReserving=True&getDailyAvailability=False&partySize=1&numEquipment=1&equipmentCategoryId=-32768 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7b0f418d7010>: Failed to establish a new connection: [Errno 101] Network is unreachable')) HOT 1
- ValidationError in RecDotGovSearchResponse due to missing description field HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from camply.