arfoll / unrarall Goto Github PK

Hello,
i just wanted to pinch in an idea while i don't have the time to do code and commit it myself in the next few days.
A "clean-misc" option to delete commonly unwanted files would be nice. List:
common OS-garbage:
.DS_Store, Thumbs.db
release files:
*.sfv (already there), $sfilename-sample.avi|sfv, $sfilename.srr|srs
Downloader dump:
*.1, *.2, *.tmp, (probably more)

If my comments are welcome, i have some more ideas i could commit in the next few weeks.

Hi,

first of all thanks for the great work ! :)

This script is really nice and works perfectly for me.

I m just wondering if there is a way to specify an output dir ? ( i red carefully the doc but seems not ... ). That would be really nice if we could and also check if the file isn't allready extracted in the output directory.

Regards.

Hey,

Thanks for the effort you guys have put in to this script! I found this script after getting to !#/bin/bash of a my own hack to do the same thing so you guys really saved me there! One thing I find odd however is that when using the following command-line:

cd /storage/download/finished; /usr/local/bin/unrarall/unrarall --clean=all /storage/download/finished;

I get the following:

Working over directory "/storage/download/finished"
Running cksfv using totally_legal_thing_to_rar.s15e06.720p.sfv
Extracting (e) "/storage/download/finished/totally_legal_thing_to_rar.s15e06.720p.rar"...ok
Running hooks...covers_folders nfo osx_junk rar sample_folders find: `./Sample': No such file or directory
sample_videos windows_junk empty_folders Finished running hooks

Everything works just as expected but the "find: `./Sample': No such file or directory" seems out of place. The rest of the feedback is from the script but that one thing comes from find itself.

From the start there is indeed a "Sample" folder which contains a sample video file and the sample file and folder are deleted during the run.

Any ideas on why this is happening? The script is being run on a Raspberry PI 2, Model B and a fresh Raspbian install. I have peeked at the code but haven't found anything obvious as of yet.

Regards,
LG

PRs #34 and #30 both try to make unrarall avoid extracting already extracted files.

Issue #31 reported problems where unrarall would clobber existing rar files upon extraction.

These problems are not independent of each and so they should be tackled together.

Ideally we

• Should not clobber existing files that come from existing archives. E.g. If a.rar contains foo and b.rar contains a different foo and we use the --output flag we'll only end up with one foo file as one will be written over the other. We should avoid this.

• We should handle rar files within rar files (i.e. do recursive extract up to some bound) and should not clobber existing rar files when we do the extraction.

• We should have an option to avoid extracting already extracted files.

The solution to these problems would probably involve examining the archive contents before extracting, always doing extraction in a temporary folder and have a sensible scheme for handling clobbering that emits a warning and tries to carry on by writing the conflicting files somewhere else.

I don't have time to work on this but if anyone wants to tackle this task that would be great.

First of all, thank you for this awesome script! I appreciate it a lot :D

I just wanted to share a fix for my problem, that I'm pretty sure has to do with sed on my Mac Os X 10.10.5 installation. The rar cleanup doesn't work correctly because it only deletes the first rar of a multi rar archive, so e.g. only archive.part01.rar with all of the rest not removed.

I was able to pin down the problem to line 612:

sfilename=`echo "$sfilename" | sed 's/\.part[0-9]\+$//'`

I changed it to

sfilename=`echo "$sfilename" | sed 's/\.part[0-9]*//'`

I just googled it quickly and found something like that.
TLDR:

On OSX, sed by default uses basic REs. You should use sed -E if you want to use modern REs, including the "+" one-or-more operator.
by paxdiablo

So nothing dramatic, just thought to share my insights :)

Take care and keep up the nice work,

J

Hi,

Every time I try to execute the command ./unrarall dir

the applications gives me "working over directory "dir"
no rar files extracted

while the dir is filled with sub folders that contain rar files.

Any idea what I am doing wrong? I got UnRar installed and cksfv. Running it on Yosemite Server.
The dir is on a second installed harddrive. I applied chmod.

Kind regards,

Maxime

If the folder where the rar-files lie has a name containing spaces extraction fails with error:

cd /test/Rhap\ sodies/
unrarall -v --full-path --clean=rar .

      Cannot open /test/Rhap.rar
      File or folder not found
      No files to extract

unrar x <filename>.part1.rar in same folder works flawlessly

Hi,

Unrarring sub files goes well (needs a couple of unpack iterations). I want to automatically move IDX, SUB and SRT files from the Subs directory to the main directory of the video/rar file(s).

Can this be added? Thanks!

Leaving this as a note to myself really - but commit f00f48b broke some rar files being left behind. I'm still not entirely clear why.

Hey. Really nice tool, thx for that.
Only thing i'm missing is an Option to Trash the files instead of deleting them.
Would be really awesome if you could add such function. thx!

Hi!
I find your unrarall script potentialy very usefull, however it would be great to be able to force script to unpack files to temporary folder before placing it back in original path.

This would prevent additional software (SickBeard/CouchPotato in my case) from processing those files before they're completely unpacked.

Could you please implement one more option to command line so the script would:

1. find all rar files in (sub)dir (as it does)
2. Unrar it to temporary path
3. Move unrared file back to original location of archives
4. Make the cleanup (as it does)

?

Thanks in advance and thanks for what already have been done.

please introduce 2 further parameters:

• recursive yes/no:
  to choose if only files in current directory are used or if also subdirectories are treated
• overwrite warning yes/no:
  ask user if file can be overwritten.
  unrarall currently ALWAYS overwrites existing files bearing the same path of extracted files.

Hi, I am having the following issue. I have a bunch of "rared" DVD files, with the classical VIDEO_TS and AUDIO_TS subdirectories. The DVDs are not inside a parent folder, so when unraring a lot of them I get only two folders and VTS_01_1.VOB.1 VTS_02_1.VOB.1 VTS_01_1.VOB.2 VTS_01_2.VOB.2 and so on... This makes the files unreadable by a player or transcoding software, and you can't even easily determine from which DVD the files came. Is there a way to create an ad-hoc directory for each decompressed file? The full-path option obviously doesn't work, as the rar files have no parent folder, e.g. VIDEO_TS and AUDIO_TS are at the root level instead of "GODFATHER/VIDEO_TS" and "GODFATHER/AUDIO_TS", which would remain problem-free. Thanks in advance.

Hi there, does this support *.zip files as well as *.rar? Considering there is 7ZIP support. Thanks! Looks like an awesome tool.

I just wanted to say first that this script is amazing. Thank you very much for the time you've put into making this and also sharing it with us!

That said, I'm curious how difficult it would be to implement an option to find/extract .iso files. 7zip using the command "7z x filename.iso" will extract these files. I took a look at the code and unfortunately I'm no where near the level of understanding in bash to contribute this addition. I did look though!

Thanks in advance!

I'm planning on run a cron-job every 48 hours to mass-extract a bunch of new RAR files that will appear on my server. However, I won't be using --clean, and thus, the RAR files will continue to stay there - so I need a way to skip already extracted ones.

However, I've found the --skip-if-exists doesn't seem to want to work, and it will re-extract files, simply overwriting the already existing ones.

pr0voked@vps:/home/redacted/extracted/Dumbo.1941.1080p.BluRay.x264-LCHD# ls
lchd-dumbo1080p.mkv
pr0voked@vps:/home/redacted/extracted/Dumbo.1941.1080p.BluRay.x264-LCHD# cd /home/redacted/rared

pr0voked@vps:/home/redacted/rared# sudo bash /home/redacted/unrarall -v -o /home/redacted/extracted/Dumbo.1941.1080p.BluRay.x264-LCHD --allow-failures --full-path --skip-if-exists Dumbo.1941.1080p.BluRay.x264-LCHD
Using "/home/redacted/extracted/Dumbo.1941.1080p.BluRay.x264-LCHD/" as output directory
Detected GNU find
Using find: find
Detecting clean up hooks...
Found unrarall_clean_covers_folders
Found unrarall_clean_empty_folders
Found unrarall_clean_nfo
Found unrarall_clean_osx_junk
Found unrarall_clean_proof_folders
Found unrarall_clean_rar
Found unrarall_clean_sample_folders
Found unrarall_clean_sample_videos
Found unrarall_clean_windows_junk
Using virtual clean-up hook none
Looking for unrar...found
Using "unrar" to extract rar files
Working over directory "Dumbo.1941.1080p.BluRay.x264-LCHD/"
Install cksfv in order to get CRC checked before using unrar
Extracting (x) "Dumbo.1941.1080p.BluRay.x264-LCHD/lchd-dumbo1080p.rar"...
UNRAR 5.60 freeware      Copyright (c) 1993-2018 Alexander Roshal


Extracting from /home/redacted/rared/Dumbo.1941.1080p.BluRay.x264-LCHD/lchd-dumbo1080p.rar

Extracting  lchd-dumbo1080p.mkv                                        1%

Extracting from /home/redacted/rared/Dumbo.1941.1080p.BluRay.x264-LCHD/lchd-dumbo1080p.r00

...         lchd-dumbo1080p.mkv 

Any ideas on what I can do?
I've checked, and double-checked, that the file is extracting to the same place (ran it about 3 times). Always the same file.

I'm testing this script on a raspberry pi 4 under raspbian lite, but get this error after execute:

pi@rpi:~ $ cd /my/test/dir && /home/pi/unrarall/unrarall --clean=all --force .
Working over directory "/my/test/dir"
Running cksfv using my.test.file.sfv
mktemp: too few X's in template ‘unrarall’ Failed to create a temporary directory to handle extraction

What's going wrong?

Thanks,
Speedy

EDIT:
Solved, with root access.
With follow call works the script fine:
cd /my/test/dir && sudo /home/pi/unrarall/unrarall --clean=all --force .

Just wondered if you could add a string of code to extract.001 archives as well? Since its supported by unrar commands by default.

And maybe think about adding unzip support on the TODO list ;)

Thanks for sharing this great tool!

First o all, thank you very much for our share. I was searching something like this time ago.

I have the following error when runing the script.

line 45: grep:: command not found
�[31mInsufficient number of arguments. See script --help�[0m

Any help?

Thank you

Hi,

this is a cool project, however, I cannot get this to work on OSX. It looks like it might be bash version differences:

unrarall: line 31: 7z: value too great for base (error token is "7z")
unrarall: line 71: `clean-up': not a valid identifier

Even if I try to repair this, I get:

$ sh unrarall -c -v
-n Looking for unrar...
-e found
Using "unrar" to extract rar files
Working over directory "/Volumes/Data/Videos"
-n Extracting (e) "/Volum"...
unrarall: line 227: u rar: syntax error in expression (error token is "rar")
1 rar files found, extracted and removed

Cheers

getting an

unrarall: line 110: `clean-up': not a valid identifier

Hello,

A very useful script, thank you so much!

But is it possible, if a file is extracted in a subdirectory, to rename it to the name of the subdirectory?

Thanks

would be useful to have a simple archive password 'memory' or db, which stores (in user's homedir, in a clear file or so) successfully used passwords, that can prompt you for browsing - and/or maybe (addictional option?) automagically tries all the password in db, for unattended operations - when some password is required.

something similar to what WinRar accomplish with its password manager (but of course there is a GUI there).

Hi. Found your script and have been attempting to get it to work on a FreeBSD-based NAS (FreeNAS).

Having some trouble, the script doesn't seem to find any rar files. Sample output:

[root@freenas] /mnt/vol1/tv/show# /mnt/vol1/backup/unrarall -d -v -s --clean=all /mnt/vol1/tv/show/test
Detecting clean up hooks...
Found unrarall-clean-covers_folders
Found unrarall-clean-empty_folders
Found unrarall-clean-nfo
Found unrarall-clean-osx_junk
Found unrarall-clean-rar
Found unrarall-clean-sample_folders
Found unrarall-clean-sample_videos
Found unrarall-clean-windows_junk
Using virtual clean-up hook all
Using "echo" to extract rar files
Working over directory "/mnt/vol1/tv/show/test"
no rar files extracted

There are, of course, many subfolders in my 'test' dir, all of which contain multi-part rar archives:

[root@freenas /mnt/vol1/tv/show/test/show.S07E13.HDTV.XviD]# ls
show.s07e13.hdtv.xvid.r03
show.s07e13.hdtv.xvid.r08
show.s07e13.hdtv.xvid.r13
show.s07e13.hdtv.xvid.nfo
show.s07e13.hdtv.xvid.r04
show.s07e13.hdtv.xvid.r09
show.s07e13.hdtv.xvid.r14
show.s07e13.hdtv.xvid.rar
... etc (yes, it's a complete rar set.)

Grateful any assistance. The default shell for FreeNAS is csh, however I get the same result when I run bash and then run the script. Note that if I run without the -d switch, I get the same end result, however it does report that it's found the unrar executable:

[root@freenas] /mnt/vol1/tv/show# /mnt/vol1/backup/unrarall -v -s --clean=all /mnt/vol1/tv/show/test
Detecting clean up hooks...
Found unrarall-clean-covers_folders
Found unrarall-clean-empty_folders
Found unrarall-clean-nfo
Found unrarall-clean-osx_junk
Found unrarall-clean-rar
Found unrarall-clean-sample_folders
Found unrarall-clean-sample_videos
Found unrarall-clean-windows_junk
Using virtual clean-up hook all
Looking for unrar...found
Using "unrar" to extract rar files
Working over directory "/mnt/vol1/tv/show/test"
no rar files extracted

other miscellanea:
UNRAR 4.20 freeware Copyright (c) 1993-2012 Alexander Roshal
unrarall script version 0.4.2

I have a folder with multiple rar archieves split into parts. When I try to extract them with ./unrarall -s path/to/files I get the error "Corrupt file or wrong password". So I checked /home/user/.unrar_passwords for the correct password. This file is per line right?

Here is the output I get when running the command. Please note that when I run unrar e xxx.part01.rar it works(same password, same archieve)

pi@mmc:/path/to/files $ ./unrarall -s Dir\ with\ some\ spaces\ between/ Working over directory "Dir with some spaces between/" Extracting (e) "Dir with some spaces between/file-se.part01.rar"... Checksum error in the encrypted file /path/to/files/Dir\ with\ some\ spaces\ between/file-se.part01.rar. Corrupt file or wrong password.

Anyone got an idea?

Hi,

when I use 'all' as cleanup hook and I skip some archive due to missing (or forgotten...) password the archive gets cleaned as well, and deleted.

would be useful some option to skip file cleanup when password is not given/wrong.

thanks for all your work!

Is this possible? I would be super grateful for this feature

example:
Santa.Claus.Conquers.the.Martians.1964.DVDRip.x264-PUBLICDOMAiN/pub-scctm.rar

extracts to:
Santa.Claus.Conquers.the.Martians.1964.DVDRip.x264-PUBLICDOMAiN/pub-scctm.mkv

option would instead give you:
Santa.Claus.Conquers.the.Martians.1964.DVDRip.x264-PUBLICDOMAiN/Santa.Claus.Conquers.the.Martians.1964.DVDRip.x264-PUBLICDOMAiN.mkv

If I run this command twice

/home/user/.script/unrarall -s --clean=all /source_parent_dir

Will the second run of the script not overlap the first command?

I'm thinking in the situation, where I want a cron job to run the command every 1 minute. The first run may still be processing when the second cron kicks off

System: Linux Mint 17
Commandline: ./unrarall -d --clean=all -v --disable-cksfv

How to run cleanup over directories that dont contain rars?

Sometimes my sub folders/files never downloaded as rars. but they still have the sample and .nfo, etc. Can this script be configured to clean those up? It seems to be ignoring them right now

Once in a while I run into an issue where rar files won't get extracted due to a missing or invalid MIME header. The rar file will extract successfully using cmd line unrar however I had to comment out the below to successfully get this to run. I wasn't able to check the mime types of the invalid but valid rar files, however I'll check next time I run into this.

# check file really is a rar file if [[ ! "file --mime "${file}"" =~ $MATCHRARMIME ]]; then message error "Skipping file \"${file}\" because it does not appear to be a valid rar file." # Treat this as a failure so we exit with a non zero exit code let FAIL_COUNT=FAIL_COUNT+1 continue fi

Hi
As you could do so that when unrarall runs, if there is already one running, it does not run?
It seems that every time unrarall is executed it does it in a different way
Thank you for the help

Uhhh unrarall made me a horror situation, it made huge mess, you need to fix this issue.
Let me explain, lot of scene files have idiotic naming of base rar file and the archived sub file which has the same rar name as the base rar file, inside the *subs.rar.
In that case unrarall would unpack firstly the sub rar which would overwrite the base rar file.
Example:
name of the base rar file rep-13hoursthesecretsoldiers.1080p.bluray.x264.rar
name of the subtitle file rared with the name "rep-13hoursthesecretsoldiers.1080p.bluray.x264-subs.rar" iside of this rar were two files:
rep-13hoursthesecretsoldiers.1080p.bluray.x264.rar and
rep-13hoursthesecretsoldiers.1080p.bluray.x264.idx
rep-13hoursthesecretsoldiers.1080p.bluray.x264.rar has rep-13hoursthesecretsoldiers.1080p.bluray.x264.sub
As you can see
In my case unrarall deleted 244 base rar files which made 30% of my rared HD movies uncompleted.

There are almost 10 scene groups (REPLICA, SPARKS...) which use such naming formatting, most of them use SUB folders but on most private trackers and on most NNTP indexers uploaders decide that SUB folder were not needed and we have because of that this problem.
There are even cases that those scene groups wouldn't use rared subs inside the rared sub.

Short description:

If using a relative path with the -o flag, unrarall complains that it's not a directory. When specifying the absolute path there is no error message.

How to reproduce:

Call on unrarall with the -o flag and give it a relative path as argument.

Example:

unrarall -o Movies Downloads

Sidenote:

[[ -d Movies ]] tests as True in the Shell.

Environment details:

OS: MacOS 10.13.3
Bash version: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)

i think i set it up correctly: i've done the chmod, installed: unrar,7zip and cksfv

when i run this command:

bash unrarall -v /path/to/rar/files/

i only get this:

Detecting clean up hooks...
Found unrarall-clean-covers_folders
Found unrarall-clean-nfo
Found unrarall-clean-osx_junk
Found unrarall-clean-rar
Found unrarall-clean-sample_folders
Found unrarall-clean-sample_videos
Found unrarall-clean-windows_junk
Using virtual clean-up hook none
Looking for unrar...found
Using "unrar" to extract rar files
Working over directory "/path/to/rar/files/"
no rar files extracted

although there are severeal rar files in the folder.

would anybody be kind enough to help?