Comments (1)
The built-in Python module fails silently:
https://stackoverflow.com/questions/15344253/robotparser-doesnt-seem-to-parse-correctly
Some alternatives are:
But the Python option is not actively maintained, and the C++ option is too much work.
I'm closing the issue as "not planned" - not that important.
"""
Visited URLs should be allowed to visit according to the robots.txt file.
See also: http://www.robotstxt.org/orig.html
"""
from urllib.robotparser import RobotFileParser
from urllib.parse import urlsplit, urljoin
import pytest
from xil.boi import _BOI_URL, get_boi_url_response
ROBOTS_TXT = "robots.txt"
robots_url = urljoin("{0.scheme}://{0.netloc}".format(urlsplit(_BOI_URL)), ROBOTS_TXT)
@pytest.fixture(name="user_agent")
def fixture_user_agent() -> str:
return "*"
def test_boi(user_agent: str) -> None:
rp = RobotFileParser()
with get_boi_url_response(robots_url) as response: #
rp.parse(response.read().decode().splitlines())
assert rp.entries
assert rp.can_fetch(user_agent, _BOI_URL)
from xil.
Related Issues (20)
- Changing the Git checkout clone depth to include the latest version tag.
- Don't publish local versions (+sha).
- Save caches in the CI (mypy, black, ruff) HOT 2
- Use Hypothesis instead of a constant URL HOT 1
- Add license HOT 1
- Add pyroma to pre-commit
- Run Isort through Ruff HOT 1
- Use `read_html`'s `storage_options` in pandas 2.1.0
- Add One-Zero bank exchange data
- Run only the "live" tests on a scheduled basis HOT 2
- Register a new "live" pytest marker and mark the relevant tests
- Add an option to disable network connection in tests
- Split the pre-commit job from the tests workflow
- Run only the "live" tests in the scheduled trigger of the tests workflow
- Consider publishing to PyPI with "Trusted Publishers"
- One Zero bank test failed HOT 7
- Add "Pepper" to known BOI banks list
- Bank Hapoalim API requires a functional cookie
- Update Discount module
- Address Geoblocking
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xil.