Comments (5)
Hello @paulvav ,
schematics
looks dead to me as well and I have no good hope that they will merge this 3.10 PR soon (and 3.11 is already there for a while).
From my point of view, considering the projects that I saw when I worked at Zyte (ex-Scrapinghub) which is the company that sponsor Spidermon, I don't remember any of them that uses schematics. jsonschema is the standard for item validation there. Perhaps outside Zyte this could be the same?
I would suggest to deprecate schematics (adding a Deprecation warning for anyone using it) in the next release (1.18.0) and them remove it from Spidermon in 1.19.0. This would make the item validation pipeline much cleaner too.
Any thoughts on that @VMRuiz @Gallaecio ?
from spidermon.
I'm working on that. I'll have a PR ready soon.
from spidermon.
Hello @paulvav , @rennerocha ,
Schematics looks abandoned indeed. Even the doc warms is currently somewhat outdated. So I'm good with deprecating this eventually.
If there is enough interest on other validation tools we could think of adding them.
But that is a different topic that doesn't need to wait for this one to resolve.
from spidermon.
@VMRuiz schematics is broken starting python 3.10 and beyond. Tightly coupling spidermon with a dead project (latest update from 2021) is not a great idea for spidermon moving forward. this is preventing us from upgrading our spider's python version. Would you be open for PRs that drop support of schematics in favor of other validation libraries? If yes, any preference on which solutions to adopt?
from spidermon.
Hello @HaddadJoe, thanks for you interest on the topic!
We are open to PR both for deprecating schematics and to add new validations libraries.
They don't necessarily needs to be done simultaneously
from spidermon.
Related Issues (20)
- fix: CI is currently failing on master HOT 3
- It can be difficult to confirm current spidermon thresholds for monitors HOT 1
- Add support for lists of dictionaries in field coverage rules
- Dependencies needs some refactoring
- Update supported python versions
- Support different severity levels for Spidermon alerts HOT 2
- Error when using custom job tags action class HOT 1
- Create ReadTheDocs configuration build file
- Field coverage always 100% HOT 5
- spidermon.utils.zyte.client uses only project setting module for settings.
- Error with SendSmtpEmail: 'NoneType' object has no attribute 'bio_read' HOT 5
- Support restricting the coverage monitor fields HOT 4
- Unable to validate date and date-time with jsonschema HOT 2
- CI is broken in master HOT 1
- Add handling for datetime aware start time in PeriodicExecutionTimeMonitor HOT 1
- Change format of content of _validation field HOT 1
- Periodic item count monitor crashes if it runs before item_scraped_count exists in stats
- Field names containing "_items" crash field coverage
- Default email template fails with latest `scrapinghub` client version HOT 1
- `SPIDERMON_MONITOR_SKIPPING_RULES` isn't documented
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spidermon.