Comments (5)
That shouldn't be possible, it seems like a bug. Does this happen often?
from durable-queue.
Yes, it does happen often.
Here's some detail of how I am using DQ. All actors working concurrently. Before, and after, the following block, there's only a single thread of execution, which pre-populates the first queue with tasks.
[DQ::some-in-q] --multiple-consumers--> [DQ::some-out-q] --single-consumer-to-drain-writing-to-a-file-as-tasks-come.
Now that you say it is unexpected, I have a question - should "put!"s be protected with a lock?
I've been having some troubles, so I introduced locking (around interval-task-seq calls) and saw some improvement - I'm sorry though, I didn't do a thorough job at investigating and recording. I'm going to try locking around the put!s too and re-run a few times.
If you'd like me to try any specific steps, do let me know.
from durable-queue.
A few more observations.
- I stopped seeing those negative numbers once I introduced (locking ...) around my operations on DQ.
- With many concurrent writes, and fsync-put disabled, I started to see garbled content when consuming from the queue being written to. I have always thought of fsync as protection against program crash, not about reliable writing to the queues. I introduced calling fsync (with the surrounding locking) for every write to this queue which is concurrently written to by 10s of threads. It looks like the problem of garbled content is now gone. BUT I'd need to look for some more time.
- The garbled content problem appeared when I pushed up the writing threads to 100. But by introducing fsync, and locking, I have also (unintentionally) introduced threads-idling as they wait to write before continuing. So, I can't tell with confidence if the locking+fsync combo solves the problem, or just makes it hard for the problem to surface. From < 10 minutes, the run-time for this dataset I am working with grew to ~30 minutes after this change.
Note: The data I am processing is being retrieved over the net (HTTP calls), and I store the raw responses in another file. For the garbled content, I did cross-check the HTTP responses and the data in the raw form appeared to be right. That's my reference point.
from durable-queue.
Well, in theory locking shouldn't be necessary on your end. I'll take a look.
from durable-queue.
Could be related to #16, I observed the same things.
from durable-queue.
Related Issues (14)
- Update to Clojure 1.8 - byte-streams 0.2.2 HOT 4
- Permissions Issues on /tmp/* Directories HOT 3
- Use cases HOT 1
- Thread safety of take!, put! HOT 4
- Slabs can grow larger than max-queue-size
- Less durable queue HOT 5
- on put! java.lang.NoSuchMethodError: java.util.zip.Checksum.update([B)V HOT 5
- Java 13 compatiblity HOT 12
- calling complete! on already completed task messes up stats HOT 2
- put! always evaluating to nil? HOT 3
- Dashes converted to underscores HOT 1
- Deletes files that look like slabs HOT 1
- names with . characters do not save properly. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from durable-queue.