Comments (3)
If tasks are getting stuck in the INITIALIZING state, it probably indicates that the scheduler (e.g. AWS Batch) is killing the jobs for some reason or another and the state update from the worker isn't making it to the database. You can turn on state reconciliation for your backend which may help:
Relevant config section:
https://github.com/ohsu-comp-bio/funnel/blob/master/config/default-config.yaml#L276-L282
Code doc:
https://github.com/ohsu-comp-bio/funnel/blob/master/compute/batch/backend.go#L140-L155
It probably wouldn't be all that hard to implement a routine that periodically scans QUEUED/INITIALIZING/RUNNING tasks and cancels them if they hit some sort of wall time specified in the config. However, it seems to me that this would just be masking an underlying issue.
from funnel.
Thanks for the pointers.
I enabled the reconciliation but I could not see any improvement (set it to check every 30m).
I occasionally have jobs that are stuck either to INITIALIZATION or RUNNING state for days (until I kill them. The only common thread I have found between those is that they are stuck at stages that require transfers of many files (e.g. >40 files) each of several GB in size.
Unfortunately, this is not reproducible, i.e. if I start the same job again it will probably go through.
I was wondering if this is really a network problem. Check for example the following plot.
It comes from a job that has finished running, and is stuck transferring files to s3 for hours. Initially it starts transferring with high speeds and then drops to a constant very slow speed. I have had similar plots for all other stuck jobs i checked.
I wonder if this is a result of trying many parallel transfers, or there is an IO block somewhere. In all these cases the funnel task is in uninterruptible sleep caused (presumably) by I/O.
from funnel.
I've added an option to the worker config to limit the number of concurrent uploads/downloads. The default value is 10.
from funnel.
Related Issues (20)
- TES v1.1: Create task with ignore_error flag
- TES: List task with next page token
- Failing lint job in tests
- github.com/golang/protobuf deprecation
- Homebrew formula returns "wrong number of arguments" error
- funnel binary not found during Slurm tests HOT 1
- Configuration for replacing path prefixes
- Boltdb deprecation HOT 2
- Task ID returned before database doc is created
- Feature Request: Add support for native K8s Executor
- Add Snakemake workflow tests to Funnel
- Funnel Tasks Fail when used as a TES Executor for Nextflow HOT 2
- The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested HOT 1
- Allow Funnel to run tasks against custom domain socket HOT 1
- Add option to connect to Gen3 systems
- Expose backend parameter to run container with additional permissions
- Funnel does not detect terminated instances on AWS
- Authentication failure connecting to mongodb HOT 1
- funnel 0.11 problems with mongodb
- funnel 0.11 error with AWS batch reconciler
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from funnel.