akalin / gopar Goto Github PK

How do you feel about supporting UTF8 filenames? par2/string.go seems to support decoding something more current than ASCII, but does not support encoding it. I don't know if there is anything in the par2 'spec' that has anything to say on the mater, but it goes without saying that this would be nice. par2cmdline does support more than ASCII at this point, so I'm having some issues with gopar where par2cmdline succeeds.

I'm running into the problem that parity files created for a file in a subdirectory seem to always test as irrepairable. Also when I change the working dir to it and remove any path from the arguments I pass. If I create parity without paths (so I chd to where the file is located) it works fine (in the dir, or somewhere else.

Have you seen this? There is an abspath check (why by the way), but that isn't being triggered (obviously). Maybe it is related?

I've written a GUI for bulk parity operations, par2deep. Since par2cmdline is quite slow, and a bit hard to compile, especially cross-platform, I'm quite happy to have found gopar, which seems a significant speed improvement!

I'm modifying par2deep to work with gopar, which means using only options that both gopar and par2cmdline support. One problem: return values. par2cmdline and derivatives have defined return values, which I use in par2deep to inform the user and propose actions. gopar seems to be no more precise than success (retval = 0) or failure (retval = -1). I've been browsing the codebase but even though I see some error reporting going on, these seem to not map 1:1 to par2cmdline. Would you be interested in adding this, or giving me some pointers? Differentiating repairable and irrepairable damage is the most important one.

For completeness sake, here the possible return values in par2cmdline:

	SUCCESS = 0 # "Success.", #can mean no error, but also successfully repaired!
	REPAIRABLE = 1 # "Repairable damage found.",
	IRREPAIRABLE = 2 # "Irreparable damage found.",
	INVALIDARGS = 3 # "Invalid commandline arguments.",
	UNUSABLE = 4 # "Parity file unusable.",
	FAIL = 5 # "Repair failed.",
	IO = 6 # "IO error.",
	INTERNAL = 7 # "Internal error",
	OOM = 8 # "Out of memory.",

The second thing I'd like to do is make gopar suitable for use as a library. By that I don't mean all the primitives it contains, but (also?) the interface through main(). The fastest way to to it would be to turn main() into par(args []string), and have main() call it. However I'm certain you would prefer a break up into separate create(), verify() and repair() functions :) In which case, the flagsetting part of main needs to be duplicated. Copy paste or is there a cleaner way? With my C++/Python hat on, I think of hiding all that in a class, but that doesn't seem to be the Go way so I ask.

• What form would you prefer? Maybe another?
• Where in the file hierarchy would you put the lib, meant for external use? Is there a convention like /cmd? Maybe just the root?

Hi, thanks for this interesting new PAR1/2 implementation!

Trying it out, it seems like the par executable tries to parse supplied options as inputs. For example:

./par c -c 5 test.par2 par
[1/3] Loading data file "5" failed: open 5: no such file or directory
panic: open 5: no such file or directory

goroutine 1 [running]:
main.main()
        ~/go/src/github.com/akalin/gopar/cmd/par/main.go:279 +0xd60

I've tried placing the -c 5 parameter elsewhere, as well as using -c5 but it seems to still fail in a similar way.

I'm compiling with Go v1.10rc2 if that makes any difference.

I'm trying to restore some corrupted backup files and since par2cmd doesn't seem to work (see this bug report ) I've tried par2go, however the process fails with an error:

andrea@marcopolo:~/par2-repair$ ~/go/bin/par v /mnt/casa/storage/1/Full-0005
[0] Loading volume file "/mnt/casa/storage/1/Full-0005" failed: unexpected ID string
panic: unexpected ID string

goroutine 1 [running]:
main.main()
	/home/andrea/go/src/github.com/akalin/gopar/cmd/par/main.go:426 +0xf7c

For larger files (the threshold is somewhere between 9.2 and 77MB) I consistently get this error when I try to create parity. Looking at memory usage, all files (one is 2.7GB) seem to be loaded in full. The error seems to come right after loading:

[1/1] Loaded data file "bsc.tar.zst" (578352090 bytes)
panic: too many data shards

goroutine 1 [running]:
main.main()

Hello!

First off, I'm so glad this library exists. I've been looking around for a pure-go par2 implementation for ages and recently came across this project. Nice work!

In my testing locally, it appears as though this library will load all file data into memory for processing. While that may be fine for a smaller dataset, my use case is in the 10s of gigabytes and obviously won't work.

I know this is likely still under active development, but would you consider having the API be stream-based? In a perfect world, I'm imagining everything being based on io.Reader and io.Writer interfaces. That way things can be processed in chunks, and a nice advantage is the source and destination streams aren't limited to being on-disk.

I actually would love to be able to hook this up to a virtual filesystem via the new io/fs package coming in go 1.16.

I'll try to take a stab at this, but I won't have much time to work on it until March or later. I mostly wanted to get this open to see if it was already planned work or not?

Thanks again!