Comments (6)
I'm gonna have a crack at adding this feature.
from spidermon.
Hello @curita !
From your description, I understood that to make this possible, we will need to create some kind of hierarchy between the actions. We will also need a way to identify if an action failed or not.
In my opinion, this will demand a big change in how we define the actions in our monitor suites and the use seems to be very specific to a very particular scenario (Sentry down) that shouldn't happen frequently. But if you have some code proposal that will allow this and keep backward compatibility, we can review it and discuss more.
Sentry is a very stable system (almost no downtime in the last month according to https://status.sentry.io/), so it looks weird that we have a number of downtime occurrences that are not negligible for our action execution.
Perhaps the implementation of SendSentryMessage
action is not robust enough and is failing even if Sentry is not down?
from spidermon.
Hello @rennerocha ,
Maybe we could implement this as an Action itself?
The Composed Action receives a list of other Actions and try to execute the first one. If any exception is raised the next one if the list is executed. If all actions in the list failed then the Action itself is marked as failed.
By combining these Composed Actions we can even create complex workflows. For example, the list of actions could be:
- Composed Action:
- Composed Actions for Notification:
- Open a Sentry ticket
- or Notify in Slack
- Composed Action for Backup:
- Dumps logs in S3
- Dumps logs in GCF
- Composed Actions for Notification:
This is maybe a too extreme case, but I'm sure there are more trivial user cases where this logic could be applied.
from spidermon.
I think this idea originated out of that Sentry downtime, but it could be nice to have for other services that can be temporarily down too (Slack, emails?).
I like the idea of Composable Actions! Though that reminds me a bit of the Compose processor for Scrapy Itemloaders, and it might be expected that all actions run, instead of using the remaining ones as a fallback. Maybe we can borrow the concept of "errbacks" instead? And have a wrapper or a base Action class where we can define an errback for when something goes wrong. That would explicitly create that hierarchy of what to call next when something breaks in an Action (at least when it's defined).
We'd have to identify when something goes wrong in an Action. I think the most basic check could catch any exception raised by the Action, and call the errback when that happens. There could be room to customize that behavior, but that could be a default.
from spidermon.
Hello @curita ,
I like the idea of Composable Actions! Though that reminds me a bit of the Compose processor for Scrapy Itemloaders, and it might be expected that all actions run, instead of using the remaining ones as a fallback. Maybe we can borrow the concept of "errbacks" instead?
You are right, naming is not my best skill. Something like errbacks
support seems much more appropiate
from spidermon.
Composed Action (or Composable Actions as @curita is better with naming than us) looks an interesting concept, and as you suggested, if we create it as a custom action, it seems that it won't require changes (at least not big ones) in how our current actions are defined and executed!
I added this issue as a possible candidate for the next release, so hopefully soon a PR will appear for review :-)
from spidermon.
Related Issues (20)
- idea: Restructure spidermon.contrib.scrapy.monitors HOT 2
- Schematics seems to no longer be supported. HOT 5
- Error when threshold values are set via ScrapyCloud settings HOT 1
- idea: Margin of error for coverage thresholds HOT 3
- idea: Percentage thresholds for UnwantedHTTPCodesMonitor HOT 6
- Spidermon fails to add `_validation` field to the item. HOT 10
- Separate Zyte related code from core
- fix: CI is currently failing on master HOT 3
- It can be difficult to confirm current spidermon thresholds for monitors HOT 1
- Add support for lists of dictionaries in field coverage rules
- Dependencies needs some refactoring
- Update supported python versions
- Support different severity levels for Spidermon alerts HOT 2
- Error when using custom job tags action class HOT 1
- Create ReadTheDocs configuration build file
- Field coverage always 100% HOT 5
- spidermon.utils.zyte.client uses only project setting module for settings.
- Error with SendSmtpEmail: 'NoneType' object has no attribute 'bio_read' HOT 5
- Support restricting the coverage monitor fields HOT 4
- Unable to validate date and date-time with jsonschema HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spidermon.