Giter Club home page Giter Club logo

unifi-protect-remux's Introduction

Go

Overview

This is a tool that converts Ubiquiti's proprietary .ubv files into standard .MP4 files with H.264 and AAC. The conversion is a remux: no transcoding takes place, the exact same video and audio essence are placed into a different container; because of this the process is reasonably fast and not CPU intensive even on a low spec ARM machine.

Native binaries are available. The easiest way (but worst performing) is to follow the "Quick Start For Ubiquiti Hardware" section. More advanced users can follow the instructions for x86 using qemu.

Please look in the github "releases" page at https://github.com/petergeneric/unifi-protect-remux/releases for builds of the latest version.

Latest Features

This release supports audio, and if FFmpeg is available then it will create MP4 files. Currently it will create one MP4 for each partition of a multi-partition .ubv, naming the files based on the date+time the video starts.

Audio muxing is new; it should account for audio/video synchronisation, however this has not been extensively tested (I don't have any camera samples where AV sync is particularly obvious). If you're experiencing issues and can supply a .ubv for me to examine please raise an issue and get in touch.

QUICK START: FOR UBIQUITI HARDWARE

Instructions for Cloud Key Gen 2 Plus (and other Ubiquiti hardware). Due to the relatively slow CPU and IO subsystems, running a remux this way will be somewhat slow. See below for instructions on how to run under Linux x86 below.

  1. Go to the "releases" page at https://github.com/petergeneric/unifi-protect-remux/releases and download the latest remux ARM64 binary (N.B. "ARM64", not "x86")
  2. Upload this to your Cloud Key home folder using SSH (SCP) with tar -xf remux-arm64.tar.gz && rm remux-arm64.tar.gz && chmod +x remux
  3. Download the latest FFmpeg ARM Static Release (available on the releases page, and also from https://johnvansickle.com/ffmpeg/)
  4. Upload this to your Cloud Key and extract it with xz -d ffmpeg-release-arm64-static.tar.xz && tar -xf ffmpeg-release-arm64-static.tar && mv ffmpeg*arm64-static/ffmpeg ./ && rm ffmpeg-release-arm64-static.tar.xz && chmod +x ffmpeg
  5. Run the following on your cloudkey: export PATH=$HOME:$PATH (you'll either need to run this every time you log in or put it in your .bashrc file)
  6. Navigate to where your .ubv video is located (base path: /srv/unifi-protect/video).
  7. Run: remux *.ubv
  8. By default, only video is extracted. If you need to extract audio too, add "--with-audio=true" to your command

If FFmpeg is not installed (or if the command fails) the remux tool will leave the raw .aac and .h264 bitstream files; these can be combined with a variety of tools.

QUICK START: FOR x86 LINUX

Dependencies: ubnt_ubvinfo from CloudKey

N.B. If you have files from a (discontinued, unsupported) x86 Protect installation, you can use the native x86 ubnt_ubvinfo tool from it - just copy that file to /usr/bin/ubnt_ubvinfo. Otherwise, the following instructions will let you run the ARM binary on your x86 machine at a slight performance penalty using QEMU. (These instructions are Ubuntu/Debian specific; pull requests welcome for CentOS/Arch equivalent)

  1. On your x86 machine, install qemu-user with: apt install -y qemu-user gcc-aarch64-linux-gnu
  2. Copy /usr/share/unifi-protect/app/node_modules/.bin/ubnt_ubvinfo from your CloudKey to /usr/bin/arm-ubnt_ubvinfo on your x86 machine
  3. Run the following:
sudo tee /usr/bin/ubnt_ubvinfo <<EOF
#!/bin/sh
export QEMU_LD_PREFIX=/usr/aarch64-linux-gnu
exec qemu-aarch64 /usr/bin/arm-ubnt_ubvinfo "\$@"
EOF
chmod +x /usr/bin/ubnt_ubvinfo

Dependencies: FFmpeg

(These instructions are Ubuntu/Debian specific; pull requests welcome for CentOS/Arch equivalent)

To install FFmpeg, run:

apt install -y ffmpeg

Extracting video

Once the dependencies are installed, use the following instructions to get the unifi-protect-remux tool working:

  1. Go to the "releases" page at https://github.com/petergeneric/unifi-protect-remux/releases and download the latest remux x86_64 binary
  2. Upload this to your Linux server and extract with tar -zxf remux-x86_64.tar.gz
  3. Transfer .ubv files from your CloudKey to your x86 server (on cloudkey, .ubv files are found under /srv/unifi-protect/video).
  4. Run: remux *.ubv
  5. By default, only video is extracted. If you need to extract audio too, add "--with-audio=true" to your command

If FFmpeg is not installed (or if the command fails) the remux tool will leave the raw .aac and .h264 bitstream files; these can be combined with a variety of tools.

Command-line arguments

Usage of remux:
  -with-audio
    	If true, extract audio
  -with-video
    	If true, extract video (default true)
  -mp4
    	If true, will create an MP4 as output (default true)
  -output-folder string
    	The path to output remuxed files to. "SRC-FOLDER" to put alongside .ubv files (default "./")
  -version
    	Display version and quit
  -force-rate int
    	If non-zero, adds a -r argument to FFmpeg invocations

NOTE ON x86 WITHOUT QEMU

The quickstart instructions above show how to run the AARCH64 ubnt_ubvinfo tool shipped with Unifi Protect on your x86 hardware. This relies on qemu-user. If this tool is not available on your x86 machine (and you don't have the native x86 version of ubnt_ubvinfo -- for copyright reasons I can't supply this tool) then you will need to run the ubnt_ubvinfo command on your Ubiquiti hardware, then transfer the .ubv files along with a cached output of ubnt_ubvinfo to your x86 machine for final extraction.

See the scripts folder in the repository: these scripts generate a summary .txt file that you can pull back alongside the .ubv and run remux on your x86 machine.

FINDING SOURCE MEDIA

This tool works on .ubv files but is primarily designed to work on "0_rotating" .ubv files. You can get a list of these files with the following on your Unifi Protect system:

find /srv/unifi-protect/video -type f -name "*_0_rotating_*.ubv"

RUNNING

Entirely locally

If you have a version of ubnt_ubvinfo and FFmpeg that runs on your system (and it's on your PATH), you can simply run remux somefile.ubv

Run ubvinfo remotely and remux locally

For better performance (and lower IO and memory overhead on your Unifi Protect server) you can use ubnt_ubvinfo on your Unifi Protect system, writing the output to a .txt file on disk. A helper script is provided for this:

  1. Simply copy prepare.sh to your Unifi Protect system and then run prepare.sh *.ubv
  2. Next, transfer the .ubv and the .ubv.txt file(s) back to your main system.
  3. Finally, run the remux binary locally on the .ubv file; the tool will automatically find and use the .ubv.txt file prepared on your Protect system.

BUILD FROM SOURCE

Simply run "make package" to compile

DEPENDENCIES

FFmpeg

This tool extracts bare audio/video bitstreams, and relies on FFmpeg to mux them into an MP4.

You can install FFmpeg on Ubuntu with:

apt install ffmpeg

There are also several FFmpeg static builds available, see https://ffmpeg.org/download.html

unifi-protect-remux's People

Contributors

kris-sum avatar petergeneric avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

unifi-protect-remux's Issues

Extract audio

It'd be easy to extract audio, however to be useful it probably needs to be implemented at the same time as #2 because it'd be fiddly for users to mux together the audio and video streams.

"Encountered track number other than 7 or 1000"...

After encountering this remuxing one of a few .ubvs I have from 2022, hit an "Encountered track number other than 7 or 1000: 1003".

Hadn't planned on creating a bug but the comment in some relevant code says it would be worthwhile to know when this error is encountered.... if there's any additional info you're interested in that I can provide happy to do so. Certainly not blocking or anything, and only hit it one time, so please also don't hold this bug open just for this if you don't want to :).

A few items getting started

While getting started on a fresh Ubuntu 22.04.1 LTS install, I had to install go and make before being able to build the remux.go file

The ubnt_ubvinfo file I am using was retrieved from a UNVR with Protect version 2.1.2

When redirecting the executable to run as aarch64, I had to modify this line to point to the file intended to be input
WAS: exec qemu-aarch64 /usr/bin/arm-ubnt_ubvinfo "$@"
MODIFIED TO: exec qemu-aarch64 /usr/bin/arm-ubnt_ubvinfo --file="$@"

By default, the ubnt_ubvinfo file will read the .ubv file and say there there is 0 partitions found and exit
Adding two rows of ----------- PARTITION START ----------- to the beginning of the text file, makes remux behave as designed and export the video file

Error waiting for ubv: exit status 1

I have successfully remuxed ubv filed directly on my Dream Machine Pro, but the problem I have is that I'm running out of space so I need to copy those files to my Ubuntu 20.04 machine and run it in Linux.

I've copied the ubnt_ubvinfo file from my UDMPro and have placed it in /usr/bin which is in my $PATH, and I have extracted the appropriate x86 remux as well as installed ffmpeg in Ubuntu 20.04 LTS.

When I attempt to run "remux *.ubv" I get the "Analysing" line, then immediately get the "Error waiting for ubv: exit status 1" message.

Any ideas?

Tool does not process multi-partitioned UBV files

Creating official issue for linking to BountySource. Per documentation:

"Currently this tool only works with single partition files. In the future it will be expanded to work with multiple partitions, by creating multiple .mp4 files."

This issue is to correct this and allow processing on multi-partitioned UBV files.

Generate MP4s directly on Ubiquiti hardware

This ticket covers adding support for directly generating a .MP4 file.

There are static arm64 FFmpeg builds which I could call out to, or I could generate an MP4 natively using something like jcodec's MP4Muxer (although that requires analysis of the bitstream to work out frame size parameters, which is something I can currently avoid having to parse out of the bitstream)

Question

Hi Petergeneric,

I have extracted video files from the UCK-G2-Plus device and copied them to my Kali machine. When I run the program I get the following error "ubnt_ubvinfo not on PATH, nor in any default search locations!". Can you please tell me if I'm missing any files from the UCK-G2-Plus device that I need to retrieve. Is there a work around this error.

Cheers

Exit status 5?

Hi! I'm trying to remux a single ubv file as a test, but somehow it runs into some timeout waiting for input and then exits with status 5. This is the output:
2021/12/30 14:53:52 Analysing 0418D650FA9D_2_timelapse_1610037463242.ubv 2021/12/30 14:53:52 First Frame: 2021-01-07 17:05:32.802 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2020-12-15 10:29:40.066 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-01-13 06:06:18.658 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-01-13 15:49:17.059 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-01-14 03:23:38.583 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-01-21 10:30:37.033 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-01-22 03:05:35.719 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-01-31 03:04:14.298 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-02-15 17:46:04.833 +0100 CET 2021/12/30 14:53:52 First Frame: 2021-02-15 17:53:30.386 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-02-23 13:27:51.405 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-03-01 16:56:12.11 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-04-22 03:04:23.448 +0200 CEST 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-05-17 17:06:42.817 +0200 CEST 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-05-20 03:19:23.494 +0200 CEST 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-06-11 11:03:42.502 +0200 CEST 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-07-06 10:07:47.207 +0200 CEST 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-10-21 03:27:50.765 +0200 CEST 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:53:52 First Frame: 2021-11-02 03:09:31.397 +0100 CET 2021/12/30 14:53:52 Video Rate Probe: WARNING probed rate was 0 fps. Assuming timelapse file and using 1fps 2021/12/30 14:54:49 Error waiting for ubv: exit status 5
I'm trying to figure out whats happening by analysing the code. It seems to be happening in the runUbvInfo() function (ubv/ubvinfo.go) but I honestly have no clue why or where to start figuring this out further...

Instructions for copying video files using scp

Hi there, it seems that this tool would be incredibly useful after getting my hands on the video footage right from my UDM-PRO. Can you please provide documentation or instructions for using SCP to copy video footage from my UDM-PRO to my laptop or NAS?

Thanks a lot, neat tool.

No ubnt_ubvinfo tool

Hi Peter,

I searched the drive and there is no ubnt_ubvinfo tool. Is there another way to convert the files ubv files. I have the unifi-protect.crt and unifi-protect.key files, if that makes a difference.

Thanks

SpectatorNY

Cannot run remux on any .ubv-files.

Hello

I'm getting ready to replace the ssd on my UDM-SE and I found this little gem to backup my files.
I run a Debian 12 server where I have downloaded the *.ubv files.

  1. I cannot run remux - Only when I do ./remux --with-audio=true (One should run it with only remux --with-audio=true *.ubv)
  2. Whenever I try to run the command this happens:

andreas@Debian11:~$ ./remux --with-audio=true *.ubv 2023/08/11 09:52:41 Analysing 70A7410BF7BB_0_rotating_1688816522280.ubv 2023/08/11 09:52:41 Error waiting for ubv: exit status 1 andreas@Debian11:~$ rm 70A7410BF7BB_0_rotating_1688816522280.ubv andreas@Debian11:~$ ./remux --with-audio=true *.ubv 2023/08/11 09:52:50 Analysing 70A7410BF7BB_0_rotating_1688826424229.ubv 2023/08/11 09:52:50 Error waiting for ubv: exit status 1 andreas@Debian11:~$ rm 70A7410BF7BB_0_rotating_1688826424229.ubv andreas@Debian11:~$ ./remux --with-audio=true *.ubv 2023/08/11 09:52:59 Analysing 70A7410BF7BB_0_rotating_1688830290222.ubv 2023/08/11 09:52:59 Error waiting for ubv: exit status 1 andreas@Debian11:~$ nano remux

So I have deleted a lot of files that I thought was corrupted or whatever, but this error contiunes to every file. So there is a user error here somewhere.

panic: runtime error: integer divide by zero

Hi,
Doing remux on an ubv file from UDM Pro I got :

2021/09/17 17:37:33 Analysing  14/68D79A8F74CB_0_rotating_1628893641320.ubv
2021/09/17 17:37:33 First Frame timestamp 2021-08-14 00:27:00.068 +0200 CEST
2021/09/17 17:37:33 First Frame timestamp 2021-08-14 00:27:00.023 +0200 CEST
2021/09/17 17:37:33 Second Frame timestamp 2021-08-14 00:27:00.023 +0200 CEST
panic: runtime error: integer divide by zero

goroutine 9 [running]:
ubvremux/ubv.extractTimecodeAndRate(0x40004d77c0, 0xa, 0xa, 0x40004d3b60, 0x5a, 0x400006c1e0)
	/opt/build-remux/unifi-protect-remux/ubv/ubvfile.go:111 +0x498
ubvremux/ubv.parseUbvInfo(0xffffc4dae09a, 0x2c, 0x40008c6080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/build-remux/unifi-protect-remux/ubv/ubvinfo.go:178 +0x354
ubvremux/ubv.runUbvInfo.func1(0xffffc4dae09a, 0x2c, 0x40008c6080, 0x400001e5d0)
	/opt/build-remux/unifi-protect-remux/ubv/ubvinfo.go:70 +0x38
created by ubvremux/ubv.runUbvInfo
	/opt/build-remux/unifi-protect-remux/ubv/ubvinfo.go:69 +0x1e8

First time I get this error. I run remux on another arm64 system than UDM, it was running fine until this file (many TB converted).

How can I debug ?

Thanks

Add native .ubv parsing

I copied all my ubv files to a different machine, only to discover ubnt_ubvinfo is needed, which introduces all sorts of ARM vs x86 issues. I'm really sure what all ubnt_ubvinfo is used for, is it possible to bypass it and just extract everything from the ubv files?

Error waiting for ubv: exit status 1

I am running remux on ubuntu 22.4
when I try to remux *.ubv I get analyzing file_name_.ubv then get follwoing error:
"Error waiting for ubv: exit status 1"

Here is the actual command from terminal:
ubuntuadmin@ubuntu01:/Downloads/FLOWER-VEG/02$ ls
68D79ACFC70E_0_rotating_1664725780179.ubv
68D79ACFCC9D_0_rotating_1664723821515.ubv
68D79AE2ADE7_0_rotating_1664718151200.ubv
68D79AE2ADEF_0_rotating_1664718037487.ubv
68D79AE3480E_0_rotating_1664724931215.ubv
68D79AE34998_0_rotating_1664721366237.ubv
68D79AE35193_0_rotating_1664722917599.ubv
ubuntuadmin@ubuntu01:
/Downloads/FLOWER-VEG/02$ remux *.ubv
2022/10/06 17:55:05 Analysing 68D79ACFC70E_0_rotating_1664725780179.ubv
2022/10/06 17:55:05 Error waiting for ubv: exit status 1

I should mention i followed these instructions: https://github.com/petergeneric/unifi-protect-remux#dependencies-ubnt_ubvinfo-from-cloudkey

What is the issue?

Error parsing track number

Following your instructions, I tried to remux the *.ubv files on a x86_64 machine (wsl ubuntu). I get the following error:

$ remux 74ACB99F8138_0_rotating_1630472266928.ubv
2021/09/11 09:25:52 Analysing  74ACB99F8138_0_rotating_1630472266928.ubv
2021/09/11 09:25:52 Error parsing track number!strconv.Atoi: parsing "

Error parsing track

Having an error when running remux on x86 using qemu. Command below. Followed latest instructions for running remux on x86 hardware and dependencies are installed.

root@homeserver:/tmp# ./remux -with-audio=true /srv/b3a0bb32-2db4-417e-b48a-69324bd7aee3/Media/Michael/unifi_video/2021/06/25/74ACB9014479_0_rotating_1624606464418.ubv
2021/08/22 20:29:00 Analysing /srv/b3a0bb32-2db4-417e-b48a-69324bd7aee3/Media/Michael/unifi_video/2021/06/25/74ACB9014479_0_rotating_1624606464418.ubv
2021/08/22 20:29:00 Error parsing track number!strconv.Atoi: parsing "--file=": invalid syntax

Wrapper might need some adjustment

Describe the bug
I followed the readme for X86 and for making the binary work on ubuntu 22.04 i actually have to call it like so:
qemu-aarch64 -L /usr/aarch64-linux-gnu /usr/bin/arm-ubnt_ubvinfo

I don't know about other systems or even other ubuntu versions, but maybe the wrapper needs some adjustments here?

Your Hardware

  • Ubuntu 22.04

Additional context
Wrapper might need a (slight?) adjustment.

panic: runtime error: slice bounds out of range [:-1]

This error was generated on x86 ver 3.0.0, with x86 version of ubnt_ubvinfo

2020/10/10 16:05:41 Analysing  /remux/ubv.ubv
2020/10/10 16:05:41 First Frame timestamp 2020-04-09 17:28:15.016 +0000 UTC
2020/10/10 16:05:41 Second Frame timestamp 2020-04-09 17:28:15.096 +0000 UTC
2020/10/10 16:05:41 First Frame timestamp 2020-04-09 17:29:18.309 +0000 UTC
2020/10/10 16:05:41 Second Frame timestamp 2020-04-09 17:29:18.389 +0000 UTC
2020/10/10 16:06:18

Analysis complete!
2020/10/10 16:06:18 First Partition:
2020/10/10 16:06:18     Tracks: 1
2020/10/10 16:06:18     Frames: 17
2020/10/10 16:06:18     Start Timecode: 2020-04-09T17:28:15Z
2020/10/10 16:06:18

Extracting 2 partitions
panic: runtime error: slice bounds out of range [:-1]

goroutine 1 [running]:
main.RemuxCLI(0xc0000ae030, 0x1, 0x1, 0x100, 0x0, 0x1, 0x4ffb74, 0x2)
        /opt/graal/unifi-protect-remux/remux.go:105 +0x1036
main.main()
        /opt/graal/unifi-protect-remux/remux.go:61 +0x2e5

Unable to run on X86 machine

Hi, I followed the instructions for running on ubuntu 20.04 (albeit on 22.04),

When running I get this error:

2023/04/12 18:59:26 Analysing E063DAAF925D_0_rotating_1677661887138.ubv
2023/04/12 18:59:26 Error waiting for ubv: exit status 1

I can't seem to work out what is going on? Any ideas? Thanks

Source file types

Not really an issue - just curiosity :)

I know the readme states "primarily designed to work on "0_rotating" .ubv files" - but - I was digging around in my CK2+ and I noted that in the same location I had both 0_rotating and also 2_rotating files.

Not exactly finding much via google - just curious if you have come across any info on the difference (as in - should I be ignoring them, or adding them to the find clause of my current script) ?

Error waiting for ubv: exit status 5

Hello, running remux on WSL2 image on Windows 10.

Verified ubnt_ubvinfo runs from command line and is in the /usr/bin folder
Verified ffmpeg is installed and runs from the command line

Running command line: ./remux *.ubv results in:
Error waiting for ubv: exit status 5

.ubv file is a_0_rotating_ source file.

Probably a permission problem right?

Simplifying automation / scripts

This isn't a bug, but an explanation of how I'm using the tool and some enhancements I've been playing with on my end. I'm trying to figure out if I'm going about this the right way, or if there's another approach.

I've got some on-site storage and my ultimate goal is to get converted MP4 files over there reliably (including downtime of either the cloudkey or the on-site storage / networking). I have some scripts that can automatically generate MP4s, and others to copy the MP4 files from the cloudkey once created, but if I delete them once backed up, the automation I have will recreate them.

I thought about doing an existence check on the destination side first, but the file naming for each of the partitions makes it so that I don't know the filenames to check ahead of time. At first, I thought about computing these filenames manually in the script. Then realized that the unix time in milliseconds of the ubv filename doesn't line up with the start time of the partition necessarily - and of course if there's more than one in a UBV, there's no way to do this based on the filename alone.

So then I decided to add a feature that creates .done in a file next to the converted MP4. This is somewhat nice, in that it allows me to skip the MP4 creation and doing the transfer. However, remux still enumerates the files on the cloudkey and does I/O to analyze the UBV partitions that I would like to prevent. So I also added a .done file on the source side once completing conversion so that I won't go back and convert again.

So after all my logic, I can run a script that scours everything on the cloudkey and quickly and efficiently skips converting UBV files if there's a corresponding .done file. In case of some issue on the cloudkey side (replacing drive, having to factory reset, etc etc...), the .done files on the destination end prevents duplicative processing if they've been converted in the past.

I know the .txt option exists, but I really don't want any of the logic for this data pipeline to have to live off of the cloudkey itself. I think my final setup will be an additional drive on the cloudkey where I put the MP4s and corresponding .done files, and I'll have my sync scripts just move the .MP4 files off of that drive (leaving the .done files) once they've been synced to their permanent location.

So with all that being said, I'd like to submit some improvements to make my workflow easier and share them with others, but I'm new to all this (scripting, Go, etc...), so I wanted to bounce my approach off others.

I've so far added a -output-listing option which creates a file marking the operation as done (and only if the MP4 itself is created). Setting this to "SRC-NAME" automatically names the file the same thing as the MP4 but instead with a .done extension.

I haven't created a new option on the source side - while I put this in the tool for now, I could probably put this in my processing script instead. Thoughts on the approach here and the behaviors I've added?

Error parsing track number!strconv.Atoi: parsing "--file=": invalid syntax

I have the same issue

user@ubuntu:~$ ./remux *.ubv
2022/01/22 02:26:49 Analysing B4FBE47EFC73_0_rotating_1559064862165.ubv
2022/01/22 02:26:49 Error parsing track number!strconv.Atoi: parsing "--file=": invalid syntax

user@ubuntu:~$ /usr/bin/ubnt_ubvinfo -P -f ./B4FBE47EFC73_0_rotating_1559064862165.ubv > output.txt
At least one file path is mandatory

output.txt:
usage: /usr/bin/arm-ubnt_ubvinfo

MANDATORY OPTIONS:
-f, --file= The path of the ubv file to be analyzed.
Can be specified multiple times.

OPTIONALS:
-h, --help Prints this help and exits.
-r, --clock-rate= When listing time components, use the specified
clock rate. If not specified, the original clock
rate will be used. For example, 1000 means that
all time components will be expressed in
milliseconds
-z, --zero-base Rebase all timestamps to 0. Ignored if the clock
rate is not specified
-t, --tid= The TID of the track to be dumped. If not
specified, all tracks will be dumped.
-p, --partitions List partitions only
-P, --partitions-boundaries Show partitions boundaries in the output
-k, --keyframes List key frames only
-H, --human-readable-wc Print the wall clock in human readable form
-s, --sync-nodes Show synchronization nodes in the output
^[[01;33m:140 Shutting down the logger leaving you in the dark. Bye bye... :(^[[0m
^[[0m
~
~
~

Add support for timelapse ubvs

At present, current tool cannot process timelapse ubv files, only rotating.

Relevant log output:

2021/01/13 03:57:40 Analysing  /remux/B4FBE49E9086_0_timelapse_1565048137805.ubv
2021/01/13 03:57:40 First Frame timestamp 2019-08-05 23:26:31.615 +0000 UTC
2021/01/13 03:57:40 Second Frame timestamp 2019-08-05 23:26:46.66 +0000 UTC
2021/01/13 03:57:40 First Frame timestamp 2019-08-09 10:23:17.664 +0000 UTC
2021/01/13 03:57:40 Second Frame timestamp 2019-08-09 10:23:32.65 +0000 UTC
2021/01/13 03:57:42

Analysis complete!
2021/01/13 03:57:42 First Partition:
2021/01/13 03:57:42     Tracks: 1
2021/01/13 03:57:42     Frames: 19901
2021/01/13 03:57:42     Start Timecode: 2019-08-05T23:26:31Z
2021/01/13 03:57:42

Extracting 2 partitions
2021/01/13 03:57:43
Writing MP4  /remux/B4FBE49E9086_0_timelapse_2019-08-05T23.26.31Z.mp4 ...
2021/01/13 03:57:43 Running:  [ffmpeg -i /remux/B4FBE49E9086_0_timelapse_2019-08-05T23.26.31Z.h264 -c copy -r 0 -y -loglevel warning /remux/B4FBE49E9086_0_timelapse_2019-08-05T23.26.31Z.mp4]
[h264 @ 0x6a85240] Stream #0: not enough frames to estimate rate; consider increasing probesize
Invalid framerate value: 0
2021/01/13 03:57:43 FFmpeg command failed! Error: exit status 1

Allow skipping partitions whose inspection/remux fails

In #37, there was a feature request to skip errors (currently remux stops on the first error). This should be an additional commandline argument, and if enabled, it should leave the temporary h264 and aac bitstream files behind so the user can manually inspect and process them if desired.

Audio Extraction not working?

I have tried a few ways, but cannot seem to get the audio extraction bit to work. If you would like I can send you a link to a ubv file.

remux F09FC213A880_2_rotating_1630289032914.ubv --with-audio=true

remux F09FC213A880_2_rotating_1630289032914.ubv -with-audio

Both produce video, however there is also an error that is thrown 'Error Waiting for ubv: exit status 2'

2021/10/31 10:39:51 Analysing --with-audio=true
2021/10/31 10:39:51 Error waiting for ubv: exit status 2

I am running this from the CKG2+.

Thanks!

BTW this tool is pretty awesome even if I cannot get the audio to work!

Extracting Frame by Frame

Tip of the hat to you as this is a FANTASTIC tool! I've a drive in hand that was pulled during operation however have confirmed the last append to the .ubv file was written to less +/- 59 seconds prior to physical removal. As a result, a large portion of the recordings are not extracted even though the cam was set for Full Time recording at 30 fps. Would it be possible to extract each frame/still from a single 1GB ubv file as the data is extremely crucial, not worried about sound at this point. Log file attached and due to the sensitivity of this am willing to compensate as well. I've tried with and without each combination of the flags available however still only exports the mp4 (with -mp4=false) Thank you!

logs_20230511.log

Timestamp unset error

First of all, great project, helped me out a lot. Now I can just have a cron job and backup onto s3 for a cloud backup.

This is the error I'm getting. I'm not sure if it's addressed or answered somewhere, ignore me if it is.
Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

Need more help / clarification

I've been fiddling around with this for a while...I got it "working" but not sure it's actually working as intended as it seems to render about 10-20 videos that are all the same even though I've remuxed literally thousands of ubv's with all different names and in different folders (ie 2021/07/01, 2021/07/04 etc).

I'm using the latest version of Ubuntu. I have a HDD that was pulled from my UDMP 2ish years ago. I pulled the ubnt_ubvinfo from that drive and per the install guide I copied it to the usr/bin folder in Ubuntu (because I don't find a usr/bin/arm-ubnt_ubvinfo folder). After installing aarch64 copying and pasting this:

sudo tee /usr/bin/ubnt_ubvinfo <<EOF
#!/bin/sh
export QEMU_LD_PREFIX=/usr/aarch64-linux-gnu
exec qemu-aarch64 /usr/bin/arm-ubnt_ubvinfo "$@"
EOF
chmod +x /usr/bin/ubnt_ubvinfo

It appears to do nothing. Am i supposed to run those 1 line at a time? I'm not a linux user by any means so perhaps I'm doing that wrong. When I try and remux after that I get a exit status 1. I also noticed that after running the above code that #1 the ubnt_ubvinfo file shrank from 4mb down to 110k. Additionally there is no arm-ubnt_ubvinfo that I can find in the usr/bin folder. I'm going mad here lol

Newb question here

I followed you instruction for the x86 and I didn't run into any errors for the most part. I checked to insure that qemu is installed on the system I want to run this tool on and it is. I also verified that the script was actually made and written to my bin directory but when I try and run remux *.ubv it says "Command 'remux' not found". Is there a way to determine why it doesn't recognize it as a command? Or more importantly where is this command set? is it a bashrc entry I need to make? Also in this step here :
Copy /usr/share/unifi-protect/app/node_modules/.bin/ubnt_ubvinfo from your CloudKey to /usr/bin/arm-ubnt_ubvinfo on your x86 machine
I assume I am renaming the file i pulled from my Dream machine pro from "unbt_ubvinfo" to "arm-ubnt_ubvinfo". Apologies in advance, Im not a seasoned Linux veteran.

Could not find codec parameters for stream 0 (Video: h264, none): unspecified size

Hello,

trying on a CloudKeyGen2plus with the handy installation remux and ffmpeg binaries, giving a single ubv file of 1GB.
the main conversion from ubv seems to be fine (it's complaining during the fps probe but looks like it found out the correct 30 fps) then tries to extract 35 partitions:

Despite a nurmerous messages like
[h264 @ 0x3d6e0c0] decode_slice_header error
[h264 @ 0x3d6e0c0] no frame!
[h264 @ 0x3d6e0c0] non-existing PPS 0 referenced

the first one is OK (viewable through windows) while the second one failed with:
[h264 @ 0x3d6e0c0] decode_slice_header error
[h264 @ 0x3d6e0c0] no frame!
[h264 @ 0x3d6d5d0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[h264 @ 0x3d6d5d0] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[mp4 @ 0x3dac890] dimensions not set
[out#0/mp4 @ 0x3dac7c0] Could not write header (incorrect codec parameters ?): Invalid argument
[vost#0:0/copy @ 0x3dad0e0] Error initializing output stream:

then remux exits with:
2023/08/05 00:07:13 FFmpeg command failed! Error: exit status 1

a .h264 file exists but the .mp4 is 0B size.

I attached a dump of the output (not sure i can generate more logs with a verbose-like options)

Is there anything I did wrong ?

logs_20230805.txt

Video resolution lower than recording

Describe the bug
I do record either in 1080p or higher (depends on the cam support), however when i copy the ubv file from the nvr and remux it the resolution i get is 640x360.

Command line arguments to reproduce
cmd: remux D021F992C06E_2_rotating_1666352559281.ubv

Output

2022/10/21 16:03:02 Analysing  D021F992C06E_2_rotating_1666352559281.ubv
2022/10/21 16:03:02 First Frame: 2022-10-21 13:42:29.38 +0200 CEST
2022/10/21 16:03:02 Video Rate Probe: File appears to be 24 fps. Use -force-rate if incorrect.
2022/10/21 16:03:03 First Frame: 2022-10-21 15:22:07.876 +0200 CEST
2022/10/21 16:03:03 Video Rate Probe: File appears to be 24 fps. Use -force-rate if incorrect.
2022/10/21 16:03:03 

Analysis complete!
2022/10/21 16:03:03 First Partition:
2022/10/21 16:03:03 	Tracks: 1
2022/10/21 16:03:03 	Frames: 84482
2022/10/21 16:03:03 	Start Timecode: 2022-10-21T13:42:29+02:00
2022/10/21 16:03:03 

Extracting 2 partitions
2022/10/21 16:03:03 
Writing MP4  ./D021F992C06E_2_rotating_2022-10-21T13.42.29+02.00.mp4 ...
2022/10/21 16:03:03 Running:  [ffmpeg -i ./D021F992C06E_2_rotating_2022-10-21T13.42.29+02.00.h264 -c copy -r 24 -y -loglevel warning ./D021F992C06E_2_rotating_2022-10-21T13.42.29+02.00.mp4]
[mp4 @ 0x55fcc6528f80] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2022/10/21 16:03:04 
Writing MP4  ./D021F992C06E_2_rotating_2022-10-21T15.22.07+02.00.mp4 ...
2022/10/21 16:03:04 Running:  [ffmpeg -i ./D021F992C06E_2_rotating_2022-10-21T15.22.07+02.00.h264 -c copy -r 24 -y -loglevel warning ./D021F992C06E_2_rotating_2022-10-21T15.22.07+02.00.mp4]
[mp4 @ 0x55bc0c4f9a80] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

ubnt_ubvinfo output
output.txt

Your Hardware

  • Ubuntu 22.04

Remux : Command Not Found

Running on Unifi OS - Cloud Key G2+ v2.1.11

Copied remux-arm64.tar.gz and ffmpeg-release-arm64-static.tar.xz into the /home directory on the box using WinSCP.

Extracted using the following commands

tar -xf remux-arm64.tar.gz && rm remux-arm64.tar.gz && chmod +x remux

xz -d ffmpeg-release-arm64-static.tar.xz && tar -xf ffmpeg-release-arm64-static.tar && mv ffmpeg*arm64-static/ffmpeg ./ && rm ffmpeg-release-arm64-static.tar.xz && chmod +x ffmpeg

Ran

export PATH=$HOME:$PATH

Then navigated to where the UBV files are and tried to run remux .ubv and got the response...

-bash: remux: command not found

Also navigated to the /home directory and tried to run it again from there with the full file path to the UBV file and still received the same message.

HELP....what have I done wrong please?

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.