Non deterministic crashs | SIGABRT about celex4-opalkelly HOT 1 OPEN

A year later but i can provide some help.

I've seen this issue manifast itself in different ways double free or corruption (!prev) being the most common but i've also seen the same problem resulting in the following backtrace.

0x00007ffff7b8d4a4 in FPGADataReader::isIMUData (data=0xbef8cfb5000c6d44 <error: Unable to access memory at address 0xbef8cfb5000c6d44>) at src/eventproc/fpgadatareader.cpp:139 139 if ((data[0] & 0x80) == 0x80 && (data[1] & 0x80) == 0 && (data[2] & 0x80) == 0x80) (gdb) bt #0 0x00007ffff7b8d4a4 in FPGADataReader::isIMUData(unsigned char*) (data=0xbef8cfb5000c6d44 <error: Unable to access memory at address 0xbef8cfb5000c6d44>) at src/eventproc/fpgadatareader.cpp:139 #1 0x00007ffff7b8f807 in FPGADataProcessor::processData(unsigned char*, long) (this=0x5555560709c8, data=0xbef8cfb5000c6d44 <error: Unable to access memory at address 0xbef8cfb5000c6d44>, length=1206460609801828) at src/eventproc/fpgadataprocessor.cpp:188 #2 0x00007ffff7b9e74c in DataProcessThread::run() (this=0x5555560708e0) at src/eventproc/dataprocessthread.cpp:94 #3 0x00007ffff7bb41b0 in XThread::staticThreadFunc(void*) (args=0x5555560708e0) at src/base/xthread.cpp:178 #4 0x00007ffff79186db in start_thread (arg=0x7fffcac1e700) at pthread_create.c:463 #5 0x00007fffe715d88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

After some debugging i found that it is because there is unsafe access to the data queue from multiple threads which causes a corrupted queue which can result in any of these errors.

Background Thread

The dataprocessthread is created and started during instantiation of Celex4 object.

popping elements from m_queueData here, whilst in background thread here.

Main GUI Thread

pipeOutFPGA is called from the main Qt Event Loop ( main thread ) which adds data to the DataQueue.

So we have 2 different threads, adding and popping data to a queue, which eventually results in corrupt data.

Here is my patch that uses a lock to protect m_queue and m_size in dataqueue.cpp from unsafe thread access.

diff --git a/CeleX/base/dataqueue.cpp b/CeleX/base/dataqueue.cpp
index c27da6d..e262bb7 100644
--- a/CeleX/base/dataqueue.cpp
+++ b/CeleX/base/dataqueue.cpp
@@ -34,18 +34,26 @@ void DataQueue::push(unsigned char *pData, long length)
     DataInfo dataIn;
     dataIn.pData = pData;
     dataIn.length = length;
+    std::unique_lock<std::mutex> mlock(mutex_);
     m_queue.push(dataIn);
     m_size += dataIn.length;
+    mlock.unlock();
+    cond_.notify_one();
 }
 
 void DataQueue::pop(unsigned char *&pData, long *length)
 {
-    if (m_queue.size() <=0)
-        return;
+    std::unique_lock<std::mutex> mlock(mutex_);
+    while (m_queue.empty())
+    {
+      cond_.wait(mlock);
+    }
 
     DataInfo dataOut = m_queue.front();
     m_queue.pop();
     m_size -= dataOut.length;
+    mlock.unlock();
 
     if (!dataOut.pData)
     {
@@ -60,12 +68,14 @@ void DataQueue::pop(unsigned char *&pData, long *length)
 
 unsigned long DataQueue::size()
 {
+    std::unique_lock<std::mutex> mlock(mutex_);
     return m_size;
 }
 
 void DataQueue::clear()
 {
-    while (m_queue.size() > 0)
+    std::unique_lock<std::mutex> mlock(mutex_);
+    while (!m_queue.empty())
     {
         DataInfo dataToDelete;
         dataToDelete = m_queue.front();
diff --git a/CeleX/base/dataqueue.h b/CeleX/base/dataqueue.h
index 9a1d171..11b050b 100644
--- a/CeleX/base/dataqueue.h
+++ b/CeleX4base/dataqueue.h
@@ -19,6 +19,8 @@
 
 #include <queue>
 #include <stdint.h>
+#include <mutex>
+#include <condition_variable>
 
 typedef struct DataInfo
 {
@@ -40,6 +42,8 @@ public:
 private:
     std::queue<DataInfo> m_queue;
     unsigned long        m_size;
+    std::mutex mutex_;
+    std::condition_variable cond_;
 };

Hope this helps.
Rob

from celex4-opalkelly.