celepixel / celex4-opalkelly Goto Github PK

Hi,

I am developing my first application based on the CeleX 04-S DVS sensor. I have a few questions about that.

1) Generation of the EventAccumulatedPic

According to the documentation, the brightness value from each event is taken and saved into a cv::Mat. My own implementation of this procedure:

this->ownAccumulatedMat = cv::Mat(cv::Size(768, 640), CV_8UC1, char(0));
// ...
auto eventVector = pSensorData->getEventDataVector();

for (auto e : eventVector) {
    int currRow = PIXELS_PER_ROW - e.row - 1;
    int currCol = PIXELS_PER_COL - e.col - 1;

    this->ownAccumulatedMat.at<unsigned char>(currRow, currCol) = (unsigned char) e.brightness;
}

results in a visible difference compared to the result provided by the SDK (cv::Mat currentSdkEventAccumulatedMat = pSensorData->getEventPicMat(EventAccumulatedPic); ) . My generated frame seems significantly noisier. Can you explain me the difference in the generation of this accumulated Mat?

2) Comparison with the threshold

The documentation of the threshold-function (CeleX4::setThreshold) indicates that the triggering of events is determined by this value. So I'm assuming that each event has a absolute difference to the last known gray-value above this threshold-value.

Comparing the brightness of each event in the EventDataVector to the corresponding pixel in (previous, current and own) EventAccumulatedPic shows, however, that this is not necessarily the case:

Here the example output of a comparison (matching / not matching):

// [...]
previousSdkAccumulatedMat: 24327 / 104185 (18.9297%)
currentSdkEventAccumulatedMat: 18575 / 109937 (14.4539%)
ownAccumulatedMat: 11476 / 117036 (8.92991%)

---

previousSdkAccumulatedMat: 26141 / 103533 (20.159%)
currentSdkEventAccumulatedMat: 18908 / 110766 (14.5812%)
ownAccumulatedMat: 13679 / 115995 (10.5488%)
// [...]

Is my source for the difference calculation wrong?

For a better understanding, I have attached a minimal code example that clarifies my questions and my calculations:

int main(int argc, char **argv) {
    CeleX4 *pCelex = new CeleX4;
    // ...
    pCelex->openSensor("");
    pCelex->setFpnFile(FPN_PATH);
    // ...
    pCelex->setEventFrameTime(60);
    pCelex->setOverlapTime(0);
    pCelex->setThreshold(30);
    // ...
    pCelex->setSensorMode(EventMode);
    auto *pSensorData = new SensorDataObserver(pCelex->getSensorDataServer(), pCelex);
    // ...
    while (true)
    {
        pCelex->pipeOutFPGAData();
        usleep(1000 * 2);
    }
}

SensorDataObserver::SensorDataObserver(CX4SensorDataServer *pServer, CeleX4 *sensor) {
    m_pServer = pServer;
    m_pCelex = sensor;
    m_pServer->registerData(this, CeleX4DataManager::CeleX_Frame_Data);

    this->ownAccumulatedMat = cv::Mat(cv::Size(768, 640), CV_8UC1, char(0));
}

void SensorDataObserver::onFrameDataUpdated(CeleX4ProcessedData* pSensorData) {
    // ...
    if (EventMode == sensorMode) {
        this->frameCount += 1;

        // ******************************************************************
        // Interpretation of applied threshold
        // Assumption: Each event should have an intensity change higher as current threshold in
        //             comparison to last known pixel intensity
        // ******************************************************************
        cv::Mat currentSdkEventAccumulatedMat = pSensorData->getEventPicMat(EventAccumulatedPic);

        auto eventVector = pSensorData->getEventDataVector();
        int diff, eventCountMatching, eventCountNotMatching;

        // 1) compare to previous 'frame' SDK accumulated data
        if (this->frameCount >= 2) {
            eventCountMatching = eventCountNotMatching = 0;
            for (auto e : eventVector) {
                int currRow = PIXELS_PER_ROW - e.row - 1;
                int currCol = PIXELS_PER_COL - e.col - 1;

                diff = (int) this->previousSdkAccumulatedMat.at<unsigned char>(currRow, currCol) - (int) e.brightness;
                if ( abs(diff) >= m_pCelex->getThreshold() ) {
                    eventCountMatching += 1;
                } else {
                    eventCountNotMatching += 1;
                }
            }
            std::cout << "previousSdkAccumulatedMat: " << eventCountMatching << " / " << eventCountNotMatching <<
                " (" << eventCountMatching / (float)(eventCountMatching+eventCountNotMatching)*100.0 << "%)" << std::endl;
        }

        // 2) compare to current SDK 'frame' accumulated data
        if (this->frameCount >= 2) {
            eventCountMatching = eventCountNotMatching = 0;
            for (auto e : eventVector) {
                int currRow = PIXELS_PER_ROW - e.row - 1;
                int currCol = PIXELS_PER_COL - e.col - 1;

                diff = (int) currentSdkEventAccumulatedMat.at<unsigned char>(currRow, currCol) - (int) e.brightness;
                if ( abs(diff) >= m_pCelex->getThreshold() ) {
                    eventCountMatching += 1;
                } else {
                    eventCountNotMatching += 1;
                }
            }
            std::cout << "currentSdkEventAccumulatedMat: " << eventCountMatching << " / " << eventCountNotMatching <<
                " (" << eventCountMatching / (float)(eventCountMatching+eventCountNotMatching)*100.0 << "%)" << std::endl;
        }

        // 3) compare to own accumulated 'frame' data
        if (this->frameCount >= 2) {
            eventCountMatching = eventCountNotMatching = 0;
            for (auto e : eventVector) {
                int currRow = PIXELS_PER_ROW - e.row - 1;
                int currCol = PIXELS_PER_COL - e.col - 1;

                diff = (int) this->ownAccumulatedMat.at<unsigned char>(currRow, currCol) - (int) e.brightness;
                if ( abs(diff) >= m_pCelex->getThreshold() ) {
                    eventCountMatching += 1;
                } else {
                    eventCountNotMatching += 1;
                }
            }
            std::cout << "ownAccumulatedMat: " << eventCountMatching << " / " << eventCountNotMatching <<
                " (" << eventCountMatching / (float)(eventCountMatching+eventCountNotMatching)*100.0 << "%)" << std::endl;
        }
        std::cout << "*********************************************************" << std::endl;

        // ******************************************************************
        // *** Difference between SDK accumulation and own implementation ***
        // ******************************************************************
        for (auto e : eventVector) {
            int currRow = PIXELS_PER_ROW - e.row - 1;
            int currCol = PIXELS_PER_COL - e.col - 1;

            this->ownAccumulatedMat.at<unsigned char>(currRow, currCol) = (unsigned char) e.brightness;
        }

        cv::imshow("currentSdkEventAccumulatedMat", currentSdkEventAccumulatedMat);
        cv::imshow("ownAccumulatedMat", this->ownAccumulatedMat);

        this->previousSdkAccumulatedMat = currentSdkEventAccumulatedMat.clone();
        char key = (char) cvWaitKey(2);
    }
    // ...
}

https://github.com/CelePixel/CeleX-SDK/blob/29f14cc5cd7fc4071dc041a977c85b6bda1893a3/Sources/SDK/eventproc/celex4.cpp#L697

Is the word "save" a typo?

Hi,

in my current project, I am considering using multiple CeleX-4 sensors - preferably on one host PC. Regarding documentation, this is generally possible:

CeleX4::ErrorCode CeleX4::openSensor(string str)
Parameters [in] str The name of CeleX Sensor
[...] When multiple Sensors work together (please configure their names before opening sensor) [...]

A first look at the method, however, shows that this parameter is not used:
https://github.com/CelePixel/CeleX4-SDK/blob/96db3cd12019e7e4b340d79361c4280462321625/Sources/SDK/eventproc/celex4.cpp#L89-L115

Have I overlooked something? And how do I configure the name of the sensor?

My current application is based on your example 'GetFrameBufferByCallback' and therefore useses a 'SensorDataObserver'. However, I had to observe random and unpredictable crashes after a longer runtime (several minutes).

I builded the CeleX SDK with debug information and traced down the error

munmap_chunk(): invalid pointer
or
double free or corruption (!prev)

in the backtrace to:

Thread 10 "frameBufferByCa" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fcbbbdaa700 (LWP 20649)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
(gdb) backtrace 
#0  0x00007fcbf903fe97 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fcbf9041801 in __GI_abort () at abort.c:79
#2  0x00007fcbf908a897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fcbf91b7b9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fcbf909190a in malloc_printerr (str=str@entry=0x7fcbf91b97a8 "munmap_chunk(): invalid pointer") at malloc.c:5350
#4  0x00007fcbf9098ecc in munmap_chunk (p=0x56047aa1cd40) at malloc.c:2846
#5  0x00007fcbf9098ecc in __GI___libc_free (mem=0x56047aa1cd50) at malloc.c:3117
#6  0x00007fcbfa473d1b in DataProcessThread::run() (this=0x56047aa17040) at ./eventproc/dataprocessthread.cpp:78
#7  0x00007fcbfa485cbf in XThread::staticThreadFunc(void*) (args=<optimized out>) at ./base/xthread.cpp:178
#8  0x00007fcbf8bb26db in start_thread (arg=0x7fcbbbdaa700) at pthread_create.c:463
#9  0x00007fcbf912288f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The specified line in the DataProcessThread::run() is actually a memory-managemend related action: delete[] data;.

But I could not discover any (obvious) mistakes in the processing chain. After some trial-and-error debugging, I suspected compiler optimizations as the problem.

I patched the SDK

--- a/CeleX/eventproc/dataprocessthread.cpp
+++ b/CeleX/eventproc/dataprocessthread.cpp
@@ -54,6 +54,8 @@ FPGADataProcessor *DataProcessThread::getDataProcessor()
     return &m_dataProcessor;
 }
 
+#pragma GCC push_options
+#pragma GCC optimize("O0")
 void DataProcessThread::run()
 {
     while (m_bRun)
@@ -80,6 +82,6 @@ void DataProcessThread::run()
                }
     }
 }
-
+#pragma GCC pop_options

to turn-off the optimizations for this function. After these changes, the program has not crashed anymore.

I'm using the gcc:

$ gcc --version
gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0

Did you ever encountered something similar?

root@ubuntu:/home/test/Downloads/CeleX4-OpalKelly/DemoGUI/Linux/CeleXDemo# sudo sh CeleXDemo.sh 
/home/test/Downloads/CeleX4-OpalKelly/DemoGUI/Linux/CeleXDemo/./CeleXDemo: error while loading shared libraries: /home/test/Downloads/CeleX4-OpalKelly/DemoGUI/Linux/CeleXDemo/./libCeleX.so.1: file too short
root@ubuntu:/home/test/Downloads/CeleX4-OpalKelly/DemoGUI/Linux/CeleXDemo# 

我在Ubuntu 16.04上安装，当我执行sudo sh CeleXDemo.sh 报错，我该怎么办

I ran into error like "Segmentation Fault" when implementing "sh CeleXDemo.sh" after I install the driver on my Ubuntu16.04, Can anyone help?

Hello,

I’m using your CeleX 4 Dynamic Vision Sensor in my current research project. When I did a frequency-based analysis of the generated events, I noticed a special ‘phenomenon’. It seems that the top rows of the sensor are more sensitive and generate more events than the other rows.

For clarification, I have performed an experiment using a homogeneous strobe light (sheet of paper as diffuser) and recorded the corresponding event stream. From this recording (in total a few seconds) I calculated the histogram of all occurred events (polarity of -1, 0 and 1).

From this histogram it is clearly visible that there is a difference of the amount of triggered events for the top rows of the sensor. Interestingly, this difference begins exactly at the transition between row 511 and 512 (2^9).

Do you have an idea why this is - or how I can avoid this?

I’m using the current FPGA configuration bitfile (‘top.bit’) from your git-repository:

$ file top.bit
top.bit: Xilinx BIT data - from top.ncd;HW_TIMEOUT=FALSE;UserID=0xFFFFFFFF - for 6slx45fgg484 - built 2018/08/08(11:45:05) - data length 0x16a78c

$ md5sum top.bit
ba7625eda7d27c2818045da47d57d262 top.bit

According to the reference of Sample User Manual, there should be an folder named Samples\Samples\lib\Linux which stores the required librarys for compiling samples, however the folder does not exist.

Is the lib folder ready to be uploaded or I can find the required library in other folder?

您好！我来自中山大学，我们实验室买了多台贵公司的CeleX4。在使用的过程中我们发现EventDenoisedBinaryPic这个功能的降噪能力十分强大。所以我们想了解一下您使用的降噪手段是哪种算法呢？

Hi,

when looking at the timestamps of the events in different configurations of the "EventFrameTime" and the "ClockRate", a question has arisen for me. I assumed that the changes to these parameters affect (among other things) the maximum achievable timestamp within a frame buffer.

Here an example for configuration:

pCelex->setEventFrameTime(60);
pCelex->setClockRate(25);
std::cout << "Used pCelex->getEventFrameTime() " << pCelex->getEventFrameTime() << std::endl;
std::cout << "Used pCelex->getClockRate() " << pCelex->getClockRate() << std::endl;
// [...]
std::cout << "\t" << "eventVector.front().t: " << eventVector.front().t << "     " << "eventVector.back().t: " << eventVector.back().t << std::endl;

The output for different configurations looks like:

Case 1
Used pCelex->getEventFrameTime() 60
Used pCelex->getClockRate() 25
eventVector.front().t: 0 eventVector.back().t: 749900

Case 2
Used pCelex->getEventFrameTime() 80
Used pCelex->getClockRate() 25
eventVector.front().t: 0 eventVector.back().t: 749908

Case 3
Used pCelex->getEventFrameTime() 60
Used pCelex->getClockRate() 35
eventVector.front().t: 0 eventVector.back().t: 749932

Case 4
Used pCelex->getEventFrameTime() 80
Used pCelex->getClockRate() 35
eventVector.front().t: 0 eventVector.back().t: 749848

My assumption for the timestamp (in a very active scene) of the last event were values around

ClockRate[Mhz] * 1000 * EventFrameTime[ms] / 2
Case 1: 25 * 1000 * 60 / 2 = 750.000
Case2: 25 * 1000 * 80 / 2 = 1.000.000
Case 3: 35 * 1000 * 60 / 2 = 1.050.000
Case 4: 35 * 1000 * 80 / 2 = 1.400.000

However, as you can see above no change occurs.

It is possible that the variable m_uiTimeStamp in the processData-Method of the FPGA-Dataprocessor is not updated correctly? Here the line

After changing m_uiTimeStamp into m_uiTimeSlice(which is updated) I get the assumed values:

Case 1: eventVector.front().t: 0 eventVector.back().t: 749968
Case 2: eventVector.front().t: 0 eventVector.back().t: 999892
Case 3: eventVector.front().t: 0 eventVector.back().t: 1049956
Case 4: eventVector.front().t: 0 eventVector.back().t: 1399880

Is my original assumption wrong?