insta360develop / mediasdk-cpp Goto Github PK

I would like to obtain the yaw, pitch and roll for each frame of the video when stitching an equirectangular video without direction lock, in order to use the orientation later to switch between facing the same direction and facing towards the same lens

The Stitcher demo app only does insp and jpg, can the SDK do DNG?

The result image has invalid panorama related exif information. The demo app outputs an error message:

[04-29 12:07:16.787] 18976 18976 I ImageAsset.cc:0069: Parse MetadataOption::Exif failed.

I have to fix it manually by the exiftool:
exiftool -ProjectionType="equirectangular" -UsePanoramaViewer="True" -FullPanoWidthPixels=6080 -FullPanoHeightPixels=3040

SDK: Linux 20210428

Will the SDK support the Insta360 X4 8K video stitching?

Hi,
I didn't see how to add the Lock Direction parameter to generate a .mp4 video using the MediaSDK.
While the Lock Direction can be set to True/False in the Insta360 Studio. See attached picture.
Do you know how to create Lock direction using the SDK ?
Or when this parameter will be available ?
Thx!

I am using this SDK to stitch insta video files(.insv files), based on the example main.cc codes. Everything works fine and the functions are normal, but I found that of the most time it is a CPU-intensive task and my GPU seems idle.

According to the source codes of main.cc in this repo, bool enable_cuda is defaulted to be true, is it the normal case that my CPU does the most work or has something gone wrong with my configuration?

My environment

• OS: Ubuntu 20.04.5 LTS x86_64
• CPU: AMD Ryzen 5 3600X (12) @ 3.800GHz
• GPU: NVIDIA GeForce RTX 2060
• g++ --version: g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
• nvcc --version:
  
• nvidia-smi:
  

How to reproduce

1. compile: g++ main.cc -std=c++11 -I../include -L../lib -lMediaSDK -lpthread -ltbb -lcuda -lMNN -o ../bin/stitcherSDKDemo
2. add to LD_LIBRARY_PATH: export LD_LIBRARY_PATH=LD_LIBRARY_PATH:/path/to/MediaSDK/lib
3. run: ./stitcherSDKDemo -inputs video1.insv video2.insv -output output.mp4

Are there any documentation regarding how the stitching works? I cant find any mathematical model or anything.

Hi,

1. Is there a plan to add support for Insta360 Pro2 camera to this SDK?
2. If yes, can you please provide timelines?
3. If no, are there any alternate recommendations to programmatically stitch images taken from Insta360 Pro2?

Thanks,
Yashwanth

The Image stitcher produces just black images.
Linux20210406/MediaSDK - demo app with -enable_flowstate argument crashes

Hi, I got below error when I run the program on EC2(g5xlarge).
The command is below.

./bin/stitcherSDKDemo -inputs ${TARGET_INPUT_INSV_00_FILE} ${TARGET_INPUT_INSV_10_FILE} -output ${OUTPUT_MP4_FILE} -stitch_type optflow -bitrate 30000000 -output_size 5760x2880

The environment is below.

• EC2(g5xlarge)
• AMI(Deep Learning AMI GPU PyTorch 1.12.1 (Ubuntu 20.04) 20221114)
• cuda10.2(docker)

The error message is below.

[02-01 05:10:40.195]  33 33 I stitcherImp.cpp:0078:  cuda size = 1 current = 0
[02-01 05:10:40.195]  33 33 I stitcherImp.cpp:0107:  gpu_count = 1 use gpu no = 0
[02-01 05:10:40.197]  33 33 V av_demuxer.cc:0285:  enable fast parse for mp4
[02-01 05:10:40.199]  33 33 V av_demuxer.cc:0301:  check is mp4 format, need filter to annexb
[02-01 05:10:40.199]  33 33 I bsf_filter.cpp:0030:  bsf_init ret:0
[02-01 05:10:40.199]  33 33 I llog.h:0316:  [MediaAsset::Load] in [3 ms]
[02-01 05:10:40.200]  33 33 I stitch_strategy.h:0071:  
 CUDA Device Info:
 Name:              NVIDIA A10G
 TotalGlobalMem:    22731M
 SharedMemPerBlock: 48M
 MaxGridSize:       1024
[02-01 05:10:40.303]  33 33 V av_demuxer.cc:0301:  check is mp4 format, need filter to annexb
[02-01 05:10:40.303]  33 33 I bsf_filter.cpp:0030:  bsf_init ret:0
curdeviceIndex = 0
[02-01 05:10:40.304]  33 33 I video_stitcherImp.cpp:0129:  ~~~~~~~~~~~~~ fps 29.97
[02-01 05:10:40.304]  33 33 I video_stitcherImp.cpp:0133:  ~~~~~~~~~~~~~ fps 29.97
[02-01 05:10:40.406]  33 33 V av_demuxer.cc:0301:  check is mp4 format, need filter to annexb
[02-01 05:10:40.406]  33 33 I bsf_filter.cpp:0030:  bsf_init ret:0
[02-01 05:10:40.406]  33 33 V decode_filter.cpp:0027:  to use hwaccel decoder:h264_cuvid
[02-01 05:10:40.476]  33 33 I codec_util.cpp:0230:  init cuda pixel format....
[02-01 05:10:40.481]  33 33 W ffmpeg:0000:  h264_cuvid @ 0x562441400e40 Invalid pkt_timebase, passing timestamps as-is.

[02-01 05:10:40.481]  33 33 V decode_filter.cpp:0037:  set decode hw_frame_ctx.
[02-01 05:10:40.642]  33 33 V encode_filter.cpp:0046:  switch to use encoder:libx264
[02-01 05:10:40.779]  33 33 V av_demuxer.cc:0301:  check is mp4 format, need filter to annexb
[02-01 05:10:40.779]  33 33 I bsf_filter.cpp:0030:  bsf_init ret:0
[02-01 05:10:40.779]  33 33 V decode_filter.cpp:0027:  to use hwaccel decoder:h264_cuvid
[02-01 05:10:40.832]  33 33 I codec_util.cpp:0230:  init cuda pixel format....
[02-01 05:10:40.837]  33 33 W ffmpeg:0000:  h264_cuvid @ 0x562445a2f400 Invalid pkt_timebase, passing timestamps as-is.

[02-01 05:10:40.837]  33 33 V decode_filter.cpp:0037:  set decode hw_frame_ctx.
[02-01 05:10:40.936]  33 33 V av_demuxer.cc:0301:  check is mp4 format, need filter to annexb
[02-01 05:10:40.936]  33 33 I bsf_filter.cpp:0030:  bsf_init ret:0
[02-01 05:10:40.936]  33 33 V event_dispatcher.cpp:0102:  libevent use phreads:0
[02-01 05:10:40.936]  33 49 V file_media_src.cpp:0299:  file media src to start.
[02-01 05:10:40.937]  33 50 V file_media_src.cpp:0299:  file media src to start.
[02-01 05:10:40.937]  33 51 V file_media_src.cpp:0299:  file media src to start.
[02-01 05:10:41.001]  33 50 E stitcher_param.hpp:0170:  blend Error
cost = 0.820287
[02-01 05:10:41.002]  33 33 I video_stitcherImp.cpp:0057:  use cancel ~~~~~~~~~~~~~~~~~~~~~~ 333333333333333333333
[02-01 05:10:41.198]  33 51 V av_demuxer.cc:0496:  ReadPacket eof.
[02-01 05:10:41.198]  33 51 V file_media_src.cpp:0071:  /tasks/insv_conversion/VID_20221102_103041_00_049.insv on end.
[02-01 05:10:41.198]  33 51 I file_media_src.cpp:0110:  estimate audio frame duration:21

I am trying to use the SDK passing color plus ON/OFF as parameter. However, when selecting ON, a path to the color model is required. Where to find a sample model?

Can we get Voice focus and Noise reduction added to the SDK?

I am using insta360 X3
I want to get feature like insta 360 app in which i can stitch and adjust frames to edit video.
How can I achieve that and which SDK is required for same

hi.

In stitcher.h, there is a comment like this:

/**
* \class ImageStitcher
* \brief A class that stitch image, only support the formats that maked by Insta360 camera, for example *.insp
*/

but! in MediaSDK-Cpp readme.md (https://github.com/Insta360Develop/MediaSDK-Cpp#supported-fileformat)
It is insp & jpg supported.

And There is also a code that checks the jpg file extension in the sample code.

like this :
std::string suffix = input_paths[0].substr(input_paths[0].find_last_of(".") + 1);
std::transform(suffix.begin(), suffix.end(), suffix.begin(), ::tolower);
if (suffix == "insp" || suffix == "jpg") {
auto image_stitcher = std::make_shared<ins_media::ImageStitcher>();
image_stitcher->SetInputPath(input_paths);

But the jpg stitch doesn't work.

cmake-build-debug/sticher
[05-23 11:47:44.303] 13360 13360 I ExtraInfoAsset.cc:0263: extrainfo asset Parse: iotool read header fail
[05-23 11:47:44.303] 13360 13360 I offset_serializer.cpp:0108: Check Offset String failed!

[05-23 11:47:44.303] 13360 13360 E OffsetUtil.cc:0872: merge offset v1v2v3 fail, offset v1 emtpy
[05-23 11:47:44.486] 13360 13360 E image_stitcherImp.cpp:0062: blender init failed

In OSC mode, insp cannot be taken, and jpg cannot be stitched. What should I do?

I am currently experiencing an issue while attempting to stitch videos using the stitcherSDKDemo.exe program within Windows Media SDK. While using Insta360 Studio for stitching, the GPU is being utilized, and the stitching process runs smoothly. However, when attempting to use Windows Media SDK's stitching program, the GPU is not being utilized. I have verified that the computer has detected the GPU card and that CUDA is not disabled. Despite this, the GPU remains inactive during the stitching process.

I have tried to resolve the issue by experimenting with multiple desktops that have different NVIDIA cards, but the issue still persists. This issue impact the performance of the stitching process, and I would appreciate any assistance in resolving this matter.

Thanks for sharing this useful sdk.
I would like to do insv conversion in parallel to reduce processing time.
Is there any way to split insv video and convert them?

@zhangyabin_nature @capjason I get a very bad result compared to Insta360 Studio. Insta360 gives:

Where as using the MediaSDK I get:

Input videos: https://www.dropbox.com/s/grbomf1vadkbozs/df.zip?dl=0
My code is simply:

std::vector<std::string> input_paths;
input_paths.push_back("VID_20171231_175657_00_002.insv");
input_paths.push_back("VID_20171231_175657_10_002.insv");
std::string output_path = "result.mp4";

STITCH_TYPE stitch_type = STITCH_TYPE::DYNAMICSTITCH;
HDR_TYPE hdr_type = HDR_TYPE::ImageHdr_NONE;

int output_width = 5760;
int output_height = 2880;
int output_bitrate = 40000000;
bool bEnableFlowState = true;

std::string strExt = input_paths[0].substr(input_paths[0].find_last_of(".") + 1);
std::transform(strExt.begin(), strExt.end(), strExt.begin(), ::tolower);
auto videoStitcher = std::make_shared<ins_media::VideoStitcher>();
videoStitcher->SetInputPath(input_paths);
videoStitcher->SetOutputPath(output_path);
videoStitcher->SetStitchType(stitch_type);
videoStitcher->SetOutputSize(output_width, output_height);
videoStitcher->SetOutputBitRate(output_bitrate);
videoStitcher->EnableFlowState(bEnableFlowState);
videoStitcher->SetStitchProgressCallback(process_callback, 0);
videoStitcher->SetStitchStateCallback(error_callback, 0);
videoStitcher->StartStitch();
getchar();
return 0;

To build the demo I had to run

g++ main.cc -std=c++11 -I../include -L../lib -lMediaSDK -lpthread -ltbb -lnppc -lnppial -lnppicc -lnppidei -lnppif -lnppig -lcuda -lMNN  -o ../bin/stitcherSDK

which is slightly updated from the readme. I'm on Ubuntu.

I use this way the command:

LD_LIBRARY_PATH=/home/jordi/Downloads/MediaSDK/lib/ ./bin/stitcherSDK -inputs /home/jordi/UNRAID/Media/Insta360/raw/VID_20230802_101523_00_028.insv /home/jordi/UNRAID/Media/Insta360/raw/VID_20230802_101523_10_028.insv -output ~/UNRAID/Media/Insta360/stitched/VID_20230802_101523_028.mp4

The output video sound is OK but the image is green constantly. What is the problem

When trying to run the program on an image taken with insta360 x3 the following error is thrown - error while loading shared libraries: libnppial.so.10: cannot open shared object file: No such file or directory

The .so files have incompatible architectures:

objdump -f ./lib/*.so | grep ^architecture
architecture: x86_64
architecture: x86_64
architecture: x86_64
architecture: x86_64
architecture: aarch64

Specifically libtbb is ARM and everything else is x86.

objdump -f ./lib/libtbb.so* | grep ^architecture
architecture: aarch64
architecture: aarch64

Hey. I'm interested in outputting the video as a sequence of images. And this -image_sequence_dir flag is supposed to do exactly that but for some reason it doesn't work. When I run the command it seems like the process is working correctly but I don't get output images at the end and no obvious issues in the terminal output. Any help is appreciated

Here's the command that I used:
.\stitcherSDKDemo -inputs "C:\zVideo\VID_20240520_124504_00_001.insv" -image_sequence_dir "C:\zVideo" -stitch_type "optiflow"

And here's the terminal output:

[06-19 18:48:03.686]  12276 2336 I ffmpeg_util.cpp:0057:  ffmpeg init
[06-19 18:48:03.686]  12276 2336 I ffmpeg_util.cpp:0193:  set ffmpeg lovel: WARN
[06-19 18:48:03.686]  12276 2336 I stitcherImp.cpp:0120:  user set  = 0
[06-19 18:48:03.700]  12276 2336 I CudaDevice.cpp:0034:
 CUDA Device Info:
 Name:              NVIDIA GeForce RTX 4060 Laptop GPU
 TotalGlobalMem:    8187M
 SharedMemPerBlock: 48M
 MaxGridSize:       1024
[06-19 18:48:03.700]  12276 2336 I stitcherImp.cpp:0100:  cuda size = 1 current = 0
[06-19 18:48:03.700]  12276 2336 I stitcherImp.cpp:0129:  gpu_count = 1 use gpu no = 0
 start stitch [06-19 18:48:03.700]  12276 2336 I video_stitcherImp.cpp:0085:  check param
[06-19 18:48:03.705]  12276 2336 I stitch_strategy.h:0071:
 CUDA Device Info:
 Name:              NVIDIA GeForce RTX 4060 Laptop GPU
 TotalGlobalMem:    8187M
 SharedMemPerBlock: 48M
 MaxGridSize:       1024
[06-19 18:48:03.791]  12276 2336 I encode_filter.h:0021:  encodec id 27
[06-19 18:48:03.791]  12276 2336 I video_stitcherImp.cpp:0135:  device index = 0
[06-19 18:48:03.791]  12276 2336 I video_stitcherImp.cpp:0150:  set filter
[06-19 18:48:03.791]  12276 2336 I video_stitcherImp.cpp:0179:  set src filter
[06-19 18:48:03.791]  12276 2336 I video_stitcherImp.cpp:0240:  init param
[06-19 18:48:03.791]  12276 2336 I demuxer.cpp:0064:  enable mjpeg demuxer? 1
[06-19 18:48:03.887]  12276 2336 V decode_filter.cpp:0027:  to use hwaccel decoder:h264_cuvid
[06-19 18:48:03.929]  12276 2336 I codec_util.cpp:0438:  init cuda pixel format....
[06-19 18:48:03.935]  12276 2336 W ffmpeg:0000:  h264_cuvid @ 0000024F2896C040 Invalid pkt_timebase, passing timestamps as-is.

[06-19 18:48:03.935]  12276 2336 V decode_filter.cpp:0037:  set decode hw_frame_ctx.
[06-19 18:48:04.590]  12276 2336 I demuxer.cpp:0064:  enable mjpeg demuxer? 1
[06-19 18:48:12.257]  12276 2336 V media_pipe.h:0044:  ~MediaPipe
[06-19 18:48:12.257]  12276 2336 V file_media_src.cpp:0038:  ~FileMediaSrc:C:\zVideo\VID_20240520_124504_00_001.insv
[06-19 18:48:12.257]  12276 2336 V codec_util.cpp:0113:  unref cuvid decode opaque.
[06-19 18:48:12.259]  12276 2336 V file_media_src.cpp:0038:  ~FileMediaSrc:C:\zVideo\VID_20240520_124504_00_001.insv

Hi,

OS: Ubuntu 18.04
Camera: Insta360 X3
I want to stitch *.insp files. For this, I am using the LinuxSDK20230621.zip which you provide at the official website. Since the library requires CUDA 10.2, I utilize this docker image.

I am able to successfully build the stitcherSDKDemo binary with the provided CMake file.
However, the following:
./stitcherSDKDemo -inputs ../data/IMG_20230802_132756_00_014.insp -output ../data/output.jpg
results in a black output image.

Terminal output:

./stitcherSDKDemo -inputs ../data/IMG_20230802_132756_00_014.insp -output ../data/output.jpg
[08-02 13:33:07.305]  1007 1007 I offset_serializer.cpp:0108:  Check Offset String failed!
[08-02 13:33:07.314]  1007 1007 I stitch_strategy.h:0071:  
 CUDA Device Info:
 Name:              NVIDIA GeForce RTX 4090
 TotalGlobalMem:    24217M
 SharedMemPerBlock: 48M
 MaxGridSize:       1024
[08-02 13:33:07.411]  1007 1007 I CudaDevice.cpp:0034:  
 CUDA Device Info:
 Name:              NVIDIA GeForce RTX 4090
 TotalGlobalMem:    24217M
 SharedMemPerBlock: 48M
 MaxGridSize:       1024

How can I obtain a correctly stitched image?
Thanks and best regards,
Thomas

The video stitched by the SDK is wrong. You can reproduce it easily with the demo application:

./stitcherSDKDemo -inputs ~/Documents/DCIM/Camera01/VID_20210416_190528_00_026.insv -output o.mp4

the output video is just a mess.

Linux_20210428

Hello

Windows10
MediaSDK20210413(windows)

stitcherSDKDemo.exe error

OpenCV(4.1.0) Error: Gpu API call (initialization error) in cv::cuda::getCudaEnabledDeviceCount, file D:\opensource\opencv4.1.0-full-source\opencv-4.1.0-src\modules\core\src\cuda_info.cpp, line 62

My pc is C-DRIVE only,
What is this message?

openssl ? ? ? 1.0.0 ?

There is no option to set the stitching parameters like (normal, dive case, custom: 0 to 6, etc).

Especially important is the custom parameter which allows manual intervention.

Hi,
I met this issue: #18, now when using the image stitcher.
Device: X3
Firmware version: v1.0.85
SDK version: LinuxSDK20231211

I'm capturing jpg images using the CameraSDK which I can confirm look legit and I'm able to view them with other software. Each jpg contains both fisheye lenses.
I managed to compile the project successfully and when running with this command (without cuda):
./StitchInstaX3 -inputs /home/l2pp/ab_l2pp/x3_stream/frames/frame_1.jpg -output /home/l2pp/ab_l2pp/x3_stream/result.jpeg
I'm getting this output:

[02-12 15:38:14.926]  2625305 2625305 I ExtraInfoAsset.cc:0263:  extrainfo asset Parse: iotool read header fail
[02-12 15:38:14.926]  2625305 2625305 E OffsetUtil.cc:0872:  merge offset v1v2v3 fail, offset v1 emtpy
[02-12 15:38:14.963]  2625305 2625305 E ImageAsset.cc:0859:  AssetFile does not contain ExtraInfo!
[02-12 15:38:14.965]  2625305 2625305 E image_stitcherImp.cpp:0096:  blender init failed

Can you please instruct on how to solve this?

lichard@ubuntu:~/Desktop/Linux_SDK/LinuxSDK/MediaSDK/example$ gcc -o StitcherSDKDemo main.cc -L../lib -lMediaSDK -lcrypto -lssl -lstdc++
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_use_certificate_chain_file@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_set_verify@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_write@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to BIO_new@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to CRYPTO_set_locking_callback@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_connect@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to BIO_clear_flags@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to BIO_set_flags@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_ctrl@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_free@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_library_init@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_ctrl@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_accept@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to ERR_get_error@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSLv23_server_method@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSLv23_client_method@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to CRYPTO_get_locking_callback@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_load_error_strings@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_new@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_use_PrivateKey_file@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to CRYPTO_num_locks@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_free@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_get_error@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_new@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_shutdown@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_set_bio@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_read@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to SSL_CTX_load_verify_locations@OPENSSL_1.0.0'
/usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to RAND_bytes@OPENSSL_1.0.0' /usr/bin/ld: ../lib/libMediaSDK.so: undefined reference to ERR_error_string@OPENSSL_1.0.0'
collect2: error: ld returned 1 exit status

你们用的openssl是1.0.0版本的，能不能提供一个你们MediaSDK编译用的依赖库出来，我自己编的openssl依然存在符号问题

I run Linux on all of my machines and would definitively purchase a One R twin if it meant I can stich my videos using the command line. Is it available to download somewhere?