[SpscArrayQueue Question] How to clear from the Producer ? about jctools HOT 7 CLOSED

Well, it allows me to answer in more than 144 chars...
The clear method is a consumer method, there is no way around that. The question then becomes can I somehow switch the consumer thread. This is possible by following these steps:

1. Consumer thread stops consuming.
2. Consumer thread correctly 'publishes' the queue to the targeted new consumer thread.
3. New consumer thread calls poll() or clear()
   Consider for example a consumer thread which reads from a queue until they observe a 'poison pill' (a special element which notifies the consumer thread it should stop consuming) they then go on to setting some volatile flag to signal the poison pill has been consumed and never take another element off the queue. Another thread than observes the flag and can call clear().

from jctools.

I appreciate all of the chars you have to spare 👍 ...

Thinking about your answer, I started to wonder now if it would be safe to call clear()from the producer thread using an SpmcArrayQueue. In my scenario I would like to clear the queue on an offer() failing and insert the element into the (then empty) queue as a next step.

Would that be an option ? Any explanations why not will be highly appreciated. Cheers.

from jctools.

Clearing the queue because it is full is an interesting requirement... you sure that's what you want to do?
If you use clear as is you will be violating the single consumer assumption, changing the consumer index concurrently with no attempt at synchronization. This can easily leave the queue in a broken state.
You could write a producer side clear() that would remove the elements from the producer side but the problem then becomes how to stop removing elements as they are being removed by the consumer thread. I can't see this working out, but maybe I'm missing something.

from jctools.

The queue is an outbound I/O stage part of an implementation of a reliable network protocol stack. If it gets full we have a slow consumer, that will get disconnected. At that point its queue will be cleared. The reconnecting client will be recovering the missing messages from another source, while we keep on producing the live flow into the original queue.

I think I understand what you are saying - that is why I was thinking about using an SpmcArrayQueue rather than an SpscArrayQueue in order to not be changing the consumer index without synchronization on calling clear() from the producer (which is acting as a second consumer at that point).

The last part refers to how to stop consumer(s) from consuming while a clear() is in progress (if I understood you correctly), which is a good point I'll have to think about, but probably I have a means of finding out as of exploiting some recovery constraints and consume and drop those elements.

Would such a producer thread performed clear() on an SpmcArrayQueue be sound ?

from jctools.

If you are planning to re-use the queue it would need clearing, otherwise there's no need. One way to implement this scheme is by the producer setting some flag for the consumer to check before poll(). This way the consumer thread can do the cleanup if required and release the queue into some reusable pool.
Calling clear() from the producer also means some random intermediate messages on the queue may still be grabbed by the consumer thread, which sounds wrong. If the consumer can detect the need for clearing than the above solution is valid.
Finally, if you are not reusing the queue you can just leave it for the GC to clear out the references instead of clearing it.

from jctools.

Thanks for your thoughts, but we really do think of needing such a synchronuous, producer-thread performed call toclear().

So sorry for being meticulous on this, but could you confirm (and sorry for me asking a second time, if you did already in the previous post), that using the multi-consumer ArrayQueue variant, a producer-thread initiated clear()would not corrupt the queue by any means ? (just for me not losing sleep over this 😃 )

(and yes, I am aware of the consumer possibly polling arbitrary messages while the clear()is still being in progress, but do think of having a way of delaing with it nicely 😏 )

from jctools.

It will work for SPMC.

from jctools.