Comments (3)
I've realised the rather obvious solution to my problem — I just fetch the first page in a blocking operation, which populates enough pages into the queue to fire up enough workers. Duh!
But still, I'd be interested to know if it's possible to keep workers idling even if the queue starts out with little or no data in it but will grow later. Or am I missing something fundamental?
from parallel.
.map is built for a fixed set of items, so its not ideal for your usecase.
i think Parallel.in_threads(4){|thread_number| .... } could be what you want, it gives you a more direct control over the threads.
from parallel.
Thanks — working directly with in_threads
seems a better solution all-round.
from parallel.
Related Issues (20)
- Rails 6.0.3: *** stack smashing detected *** HOT 3
- Process.fork is not supported by this Ruby HOT 7
- What is the recomended way to implement a semaphore or lock using this library? HOT 1
- Parallel causes a net slowdown HOT 6
- The problem with Parallel Gem and The MySQL server is running with the --read-only HOT 2
- How is it to more simply using that ActiveRecord::Base.connection.reconnect? HOT 2
- About parallel connection logic HOT 1
- ZeroDivisionError: float division by zero HOT 1
- Sidekiq + Parallel performance issues HOT 1
- Hangs indefinitely when used with EventedFileUpdateChecker HOT 5
- Use '{}' can't concat string properly HOT 1
- parallel groupby HOT 1
- net_http_args nil class exception HOT 2
- Segmentation fault HOT 10
- [feature] finish_in_order - process finish hook in the original order HOT 7
- looking for more high level documentation HOT 1
- Ruby 3.5 to drop `win32ole` HOT 2
- rubocop --parallel failing with "negative array size" HOT 15
- Parallel.map sometimes hangs for two minutes when collecting processes HOT 7
- compatibility with ruby-progressbar (and csv) HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parallel.