Comments (7)
For me, it's not because it only moves the complexity to the user (so it increases the complexity needed to use IntelMQ, and as anything made by users, it's vulnerable to human errors ;)). And in fact, it looks like partially implemented feature (´scheduled´ type of running means nothing for IntelMQ).
I propose a little different solution than celery: for bots with the
scheduled
running type, add another configuration option with the schedule definition (e.g. in cron format;intelmqctl check
should check syntax). Then, let a separate daemon be running in background, read this configuration periodically and ensure execution (e.g. using https://github.com/agronholm/apscheduler). I think it should be rather simple and flexible solution, with a possibility to even replace the scheduler with different solution (if you prefer e.g. generating crontab instead).
Thank you for the input @kamil-certat . Actually what you describe is more or less what I had in mind with Celery, bots would have a config parameter for scheduled runs. I suggested Celery mainly because it has a programmatic interface for this as well as it is battle tested.
I hope life permits and I can come up with the time soon to write the IEP
@aaronkaplan !
from intelmq.
I totally agree that this feature would make user's life much easier (also on the documentation/standardization of IntelMQ deployments). I can see that it will add more code surface to the project but solutions like Celery (pretty mature + compatible with current IntelMQ stack) for scheduling could introduce many new use-cases and features for bots.
This also means that we would not be differentiating set ups according to the existence of systemd or dockerization. Instead there would be a unified approach in terms of deployment and configuration.
from intelmq.
Not sure if I think this is good. Why?
- leave scheduling to the program which was made for it (cron , etc.) . Do not try to add that complexity to intelmq
- it adds code and complexity
But I see the need for something like that (maybe another type of cron) inside of docker.
from intelmq.
I understand your point, but I believe it - when implemented properly - would remove complexity rather than adding it. Leaving such an important feature to be manages outside the IntelMQ adds a significant complexity for users. I think it's reasonable to expect IntelMQ to handle it; on the other hand, we could also leave starting normal bots to user and do not support it through API/CLI - with just saying that it's a complexity. ;)
However, this issue is so far more a reminder - a solution should be carefully designed, keeping the simplicity in mind. You know, I suggested using Python-native solutions, but an alternative would be to e.g. configure underlying program (cron, systemd, etc.), taking away the manual task from user without deep changes.
from intelmq.
@arvchristos could you maybe sketch out how you would see an IntelMQ + celery integration? Like as in an IEP?
from intelmq.
Maybe I should be a bit more verbose why my original answer was that I was afraid of adding complexity:
because we already have a mechanism to run bots once (one-shot) only. So, if we use that config option + schedule them via cronjob, then I would say that's the simplest way of achieving that. No?
from intelmq.
For me, it's not because it only moves the complexity to the user (so it increases the complexity needed to use IntelMQ, and as anything made by users, it's vulnerable to human errors ;)). And in fact, it looks like partially implemented feature (´scheduled´ type of running means nothing for IntelMQ).
I propose a little different solution than celery: for bots with the scheduled
running type, add another configuration option with the schedule definition (e.g. in cron format; intelmqctl check
should check syntax). Then, let a separate daemon be running in background, read this configuration periodically and ensure execution (e.g. using https://github.com/agronholm/apscheduler). I think it should be rather simple and flexible solution, with a possibility to even replace the scheduler with different solution (if you prefer e.g. generating crontab instead).
from intelmq.
Related Issues (20)
- Revisit documentation on supported OS versions and packages HOT 2
- deprecate readthedocs.io
- crontab is missing shadowserver update schema script
- Shadowserver parser docs missing part on feedname HOT 2
- intelmq.lib.exceptions.KeyExists HOT 5
- Feature Request: Shadowserver parser setting feed.documentation link HOT 1
- Upgrades error HOT 9
- Can we change it to check for empty "" instead of None. Shadowserver api collector. HOT 5
- documentation build workflow fails in forks
- Initial fetch of shadowserver-schema.json fails even in auto_update mode
- On Windows 10 using Msys2 / Mingw64 - ModuleNotFoundError: No module named 'grp' HOT 3
- intelmq_gen_feeds_conf: yaml.load deprecated HOT 3
- Allow sieve reuse data from the event
- CLI command in ShadowServer API collector/parser to list available report types HOT 2
- pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure
- Shadowserver API bot HOT 8
- (doc) Documentation wrong for debian package installation
- Unhardcode paths, add more Unix support. HOT 1
- Shadowserver parser: missing documentation on auto_update
- Introduce strong formatting & linting aka black/ruff
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from intelmq.