Shallow Optical Flow Three-Stream CNN For Macro and Micro-Expression Spotting From Long Videos
「 Innovative and Passionate 」
「 An AI Engineer with great enthusiasm 」
「 Never Stop Learning 」
😄 My Pronouns: He/Him
⚡ Research Interest: Artificial Intelligence, Affective Computing, Digital Twin, Pattern Recognition
✨ Google Scholar: @profile
Expand to view
Can yo tell me about paper related with this project ? thanks
sorry,but this CAS(ME)2 - http://fu.psych.ac.cn/CASME/cas(me)2-en.php is empety,how can i get the datasets?
Hello! I'm trying to make a picture like Fig.1 in the paper, i.e., teaser.png in SoftNet-SpotME/images. Specifically, how did you make the heat-map/attention-map type pictures in the middle column and on the right? I guess the reddish parts mean higher intensity and the bluish parts indicate weaker. Is it made by placing an intensity mask onto the original picture? I would appreciate it very much if you could share the code. Thanks in advance!
Great work, is it convenient to provide a test demo？
May i know how to create own SoftNet Weight in hdf5 file ? can you explain to me or maybe you have a simple code for create it ? thanks
Thanks a lot for your work and code! I've learned a lot from that, including how to plot the peak graph. But there is no corresponding code for plotting the intervals, including Ground-truth and predictions. Did you plot those also using matplotlib? Could you please tell me how to plot those?
Can I only use cas (me) ^ 2 for training? If so, can you provide a pre-trained model？Thank you very much!
The output seems not to have any errors, but the final result is really bad.
2023-08-02 22:20:21.207936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
------ Spotting CASME_sq micro-expression -------
------ Croping Images ------
Subject s15
Subject s16
Subject s19
Subject s20
Subject s21
Subject s22
Subject s23
Subject s24
Subject s25
Subject s26
Subject s27
Subject s29
Subject s30
Subject s31
Subject s32
Subject s33
Subject s34
Subject s35
Subject s36
Subject s37
Subject s38
Subject s40
------ Loading Images ------
Subject: s15
Subject: s16
Subject: s19
Subject: s20
Subject: s21
Subject: s22
Subject: s23
Subject: s24
Subject: s25
Subject: s26
Subject: s27
Subject: s29
Subject: s30
Subject: s31
Subject: s32
Subject: s33
Subject: s34
Subject: s35
Subject: s36
Subject: s37
Subject: s38
Subject: s40
------ Loading Excel ------
------ Loading Ground Truth From Excel ------
Total Videos: 33
------ Computing k ------
k (Half of average length of expression) = 6
------ Feature Extraction & Pre-processing ------
Video 0 Done
Video 1 Done
Video 2 Done
Video 3 Done
Video 4 Done
Video 5 Done
Video 6 Done
Video 7 Done
Video 8 Done
Video 9 Done
Video 10 Done
Video 11 Done
Video 12 Done
Video 13 Done
Video 14 Done
Video 15 Done
Video 16 Done
Video 17 Done
Video 18 Done
Video 19 Done
Video 20 Done
Video 21 Done
Video 22 Done
Video 23 Done
Video 24 Done
Video 25 Done
Video 26 Done
Video 27 Done
Video 28 Done
Video 29 Done
Video 30 Done
Video 31 Done
Video 32 Done
All Done
------ Pseudo-Labeling ------
Total frames: 80463
------ Leave one Subject Out ------
Frame Index for each subject:-
Subject 0 : 0 -> 3336
Subject 1 : 3336 -> 14273
Subject 2 : 14273 -> 18638
Subject 3 : 18638 -> 19703
Subject 4 : 19703 -> 26673
Subject 5 : 26673 -> 29790
Subject 6 : 29790 -> 44942
Subject 7 : 44942 -> 47211
Subject 8 : 47211 -> 52966
Subject 9 : 52966 -> 61673
Subject 10 : 61673 -> 71111
Subject 11 : 71111 -> 72186
Subject 12 : 72186 -> 77205
Subject 13 : 77205 -> 80463
Total X: 80463 , Total y: 80463
------ SOFTNet Training & Testing ------
2023-08-03 13:44:00.084185: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2023-08-03 13:44:00.130458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: NVIDIA GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2023-08-03 13:44:00.130586: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2023-08-03 13:44:00.137630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2023-08-03 13:44:00.137722: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2023-08-03 13:44:00.144323: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2023-08-03 13:44:00.152444: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2023-08-03 13:44:00.256133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2023-08-03 13:44:00.264745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2023-08-03 13:44:00.265462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2023-08-03 13:44:00.265563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-08-03 13:44:00.266174: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-03 13:44:00.439964: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x215d9258ab0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-08-03 13:44:00.440066: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2023-08-03 13:44:00.440971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: NVIDIA GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2023-08-03 13:44:00.441142: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2023-08-03 13:44:00.441473: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2023-08-03 13:44:00.441768: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2023-08-03 13:44:00.442052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2023-08-03 13:44:00.442336: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2023-08-03 13:44:00.442652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2023-08-03 13:44:00.443101: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2023-08-03 13:44:00.443408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2023-08-03 13:44:00.443711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-08-03 13:44:05.095469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-08-03 13:44:05.095575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2023-08-03 13:44:05.096288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2023-08-03 13:44:05.100301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6573 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3070, pci bus id: 0000:09:00.0, compute capability: 8.6)
2023-08-03 13:44:05.114755: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x216aeadfba0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-08-03 13:44:05.115189: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3070, Compute Capability 8.6
Subject : 1
------Initializing SOFTNet-------
D:\Softwares\miniconda3\envs\softnet_spotme\lib\site-packages\tensorflow\python\keras\engine\training.py:1905: UserWarning: `Model.predict_generator` is deprecated and will be removed in a future version. Please use `Model.predict`, which supports generators.
warnings.warn('`Model.predict_generator` is deprecated and '
2023-08-03 13:44:06.162629: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2023-08-03 13:44:06.415811: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2023-08-03 13:44:13.414818: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2023-08-03 13:44:13.611201: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2023-08-03 13:44:26.898090: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0
2023-08-03 13:44:27.020066: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0
2023-08-03 13:44:27.508400: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
278/278 [==============================] - 25s 14ms/step
Video: 0
Video: 1
TP: 1 FP: 34 FN: 1
Done Subject 1
Subject : 2
------Initializing SOFTNet-------
911/911 [==============================] - 14s 15ms/step
Video: 2
Video: 3
Video: 4
Video: 5
Video: 6
TP: 2 FP: 134 FN: 8
Done Subject 2
Subject : 3
------Initializing SOFTNet-------
363/363 [==============================] - 4s 12ms/step
Video: 7
TP: 2 FP: 142 FN: 9
Done Subject 3
Subject : 4
------Initializing SOFTNet-------
88/88 [==============================] - 1s 13ms/step
Video: 8
TP: 2 FP: 155 FN: 12
Done Subject 4
Subject : 5
------Initializing SOFTNet-------
580/580 [==============================] - 8s 14ms/step
Video: 9
Video: 10
TP: 3 FP: 207 FN: 19
Done Subject 5
Subject : 6
------Initializing SOFTNet-------
259/259 [==============================] - 4s 14ms/step
Video: 11
Video: 12
TP: 4 FP: 238 FN: 22
Done Subject 6
Subject : 7
------Initializing SOFTNet-------
1262/1262 [==============================] - 19s 15ms/step
Video: 13
Video: 14
Video: 15
Video: 16
Video: 17
Video: 18
TP: 7 FP: 351 FN: 30
Done Subject 7
Subject : 8
------Initializing SOFTNet-------
189/189 [==============================] - 3s 16ms/step
Video: 19
TP: 7 FP: 372 FN: 31
Done Subject 8
Subject : 9
------Initializing SOFTNet-------
479/479 [==============================] - 7s 14ms/step
Video: 20
Video: 21
TP: 8 FP: 416 FN: 32
Done Subject 9
Subject : 10
------Initializing SOFTNet-------
725/725 [==============================] - 12s 16ms/step
Video: 22
Video: 23
Video: 24
TP: 9 FP: 447 FN: 37
Done Subject 10
Subject : 11
------Initializing SOFTNet-------
786/786 [==============================] - 11s 14ms/step
Video: 25
Video: 26
Video: 27
Video: 28
TP: 13 FP: 487 FN: 39
Done Subject 11
Subject : 12
------Initializing SOFTNet-------
89/89 [==============================] - 1s 14ms/step
Video: 29
TP: 13 FP: 494 FN: 41
Done Subject 12
Subject : 13
------Initializing SOFTNet-------
418/418 [==============================] - 6s 14ms/step
Video: 30
Video: 31
TP: 14 FP: 525 FN: 42
Done Subject 13
Subject : 14
------Initializing SOFTNet-------
271/271 [==============================] - 4s 14ms/step
Video: 32
TP: 14 FP: 546 FN: 43
Done Subject 14
TP: 14 FP: 546 FN: 43
Precision = 0.025
Recall = 0.2456
F1-Score = 0.0454
COCO AP@[.5:.95]: 0.0016
For training data, where are the trained weights saved? Is SOFTNet_Weights\CASME_sq\micro-expression? When I use pre-trained weight for prediction, is this weight I trained before? Thank you very much for helping me answer
xl = pd.ExcelFile(dataset_name + '/code_final.xlsx') # Specify directory of excel file
xlrd.biffh.XLRDError: File size is 0 bytes
How did this file come about? Following the previous steps will not yield this file
------ Computing k ------
Traceback (most recent call last):
File "d:/HuaweiMoveData/Users/ANJIAQI/Desktop/code/SoftNet-SpotME-main/main.py", line 63, in
main(config)
File "d:/HuaweiMoveData/Users/ANJIAQI/Desktop/code/SoftNet-SpotME-main/main.py", line 32, in main
k = cal_k(dataset_name, expression_type, final_samples)
File "d:\HuaweiMoveData\Users\ANJIAQI\Desktop\code\SoftNet-SpotME-main\load_label.py", line 98, in cal_k
N=total_duration/len(samples)
ZeroDivisionError: division by zero
总是出现这个错误，应该如何解决
Hi, thanks for your sharing.
I ran the code on CASME for micro-expression detection, and I almost reproduced the results reported in the paper (0.1212 vs. 0.1379). However, I found that the evaluation was done only on videos in which micro-expressions occured (33 vs. 97 (total number), only cover 14 subjects vs. 22 subjects (total number)). Since the model may misjudge non-expression snippets as micro-expression in the rest videos, the number of false positives may be underestimated and the results reported in the paper may be overestimated,
I have the SAMM dataset but when I keep having an error that len(samples) is zero in the program of running, is it because I only have one dataset? I also wonder can this program run on linux? Thank you very much!
May I know how to choose the best model parameters? or you just save the model after training 10 epochs?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.