mobileroboticsskoltech / recsync-android Goto Github PK

Video recording app with sub-millisecond synchronization accuracy for multiple Android smartphones, useful for creating affordable and easy-to-setup multi-view camera systems for robotics, SLAM, 3D-reconstruction, panorama stitching

License: Apache License 2.0

Java 92.48% Python 5.51% Shell 2.01%

android camera camera2-api slam synchronization multi-view 3d-reconstruction neural-rendering video

recsync-android's Introduction

If you use this application, please cite Sub-millisecond Video Synchronization of Multiple Android Smartphones:

@misc{akhmetyanov2021submillisecond,
      title={Sub-millisecond Video Synchronization of Multiple Android Smartphones}, 
      author={Azat Akhmetyanov and Anastasiia Kornilova and Marsel Faizullin and David Pozo and Gonzalo Ferrer},
      year={2021},
      eprint={2107.00987},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Usage:

Leader smartphone setup

Start a Wi-Fi hotspot.
The app should display connected clients and buttons for recording control

Client smartphones setup

Enable WiFi and connect to the Wi-Fi hotspot.

Recording video

[Optional step] Press the calculate period button. The app will analyze frame stream and use the calculated frame period in further synchronization steps.
Adjust exposure and ISO to your needs.
Press the phase align button.
Press the record video button to start synchronized video recording.
Get videos from RecSync folder in smartphone root directory.

Extraction and matching of the frames

Requirements:

- Python
- ffmpeg

Navigate to utils directory in the repository.
Run ./match.sh <VIDEO_1> <VIDEO_2>.
Frames will be extracted to directories output/1 and output/2 with timestamps in filenames, output directory will also contain match.csv file in the following format:
```
timestamp_1(ns) timestamp_2(ns)
```

Our contribution:

Integrated synchronized video recording
Scripts for extraction, alignment and processing of video frames
Experiment with flash blinking to evaluate video frames synchronization accuracy
Panoramic video demo with automated Hugin stitching

Panoramic video stitching demo

Link to youtube demo video

We provide scripts to stitch 2 syncronized smatphone videos with Hujin panorama CLI tools
Usage:
- Run ./make_demo.sh {VIDEO_LEFT} {VIDEO_RIGHT}

This work is based on "Wireless Software Synchronization of Multiple Distributed Cameras"

Reference code for the paper Wireless Software Synchronization of Multiple Distributed Cameras. Sameer Ansari, Neal Wadhwa, Rahul Garg, Jiawen Chen, ICCP 2019.

recsync-android's People

Contributors

Stargazers

Watchers

Forkers

chaoso liuguoyou asetsuna azaat dfki-signlanguage sttomato minhpvo prime-slam ipsych kaigeli minho3 gitmatih

recsync-android's Issues

How to fix the focal length of camera

Thanks for sharing this work! I can fix the focal length in the phone camera setting, but this app will call the phone camera itself. How can I fix the focal length of the camera?

Different video length from multiple phones

Hi, thanks for the work. We installed the apk and tested it using four phones (the same brand but different models). We indeed align the phases. However, we found the videos captured from different cell phones have different lengths. What's the potential reason causing the issue?

Thanks!

[RecSync] Improve README

OverflowError when using match.sh util

I get the following error trying to use the match.sh util:

Traceback (most recent call last):
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\dtypes\cast.py", line 2040, in maybe_cast_to_integer_array
    casted = np.array(arr, dtype=dtype, copy=copy)
OverflowError: Python int too large to convert to C long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./get_match.py", line 64, in <module>
    main()
  File "./get_match.py", line 23, in main
    match(vid_1, vid_2)
  File "./get_match.py", line 38, in match
    left = pd.DataFrame({'t': image_timestamps_1,
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\frame.py", line 614, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\internals\construction.py", line 464, in dict_to_mgr
    return arrays_to_mgr(
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\internals\construction.py", line 124, in arrays_to_mgr
    arrays = _homogenize(arrays, index, dtype)
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\internals\construction.py", line 589, in _homogenize
    val = sanitize_array(
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\construction.py", line 570, in sanitize_array
    subarr = _try_cast(data, dtype, copy, raise_cast_failure)
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\construction.py", line 755, in _try_cast
    subarr = maybe_cast_to_integer_array(arr, dtype)
  File "C:\Users\theez\AppData\Roaming\Python\Python38\site-packages\pandas\core\dtypes\cast.py", line 2044, in maybe_cast_to_integer_array
    raise OverflowError(
OverflowError: The elements provided in the data cannot all be casted to the dtype int32

Maybe it can be fixed using 'int64' instead of int here - at least that's how I managed to get it running.

My environment:

Windows 10 64-bit
Python 3.8.5
numpy 1.21.1
pandas 1.3.1
sys.maxsize returns 9223372036854775807 (if it is of any help)

The files (2 videos with their .csvs) that I used can be found here.

[RecSync] Separate time synchronization module w various methods

Could be based on Twist-n-Sync
GUI for method selection