How to write to an existing zstd file using io.writer? about zstd HOT 4 CLOSED

Hi @zplzpl

There are some internal buffers usually on io.Writer and files so you need to properly close the objects to make sure the full data is written:

func panicOnError(err error) {
  if err != nil {
    panic(err)
  }
}

func main() {
  filename := "./test.zstd"
  file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, 0644)
  panicOnError(err)
  defer file.Close()

  w := zstd.NewWriter(file)
  defer w.Close()
  _, err = w.Write([]byte("aaaaa"))
  panicOnError(err)
}

Hope this helps!

from zstd.

Oh, yes, I didn't do the correct closing operation.

I did what you said and it works.

But I have a problem, if the process crashes and exits, there is no correct close operation. Then there will be problems with continuing to write.

Is there any way to judge this zstd file, whether it was normally closed before?

If not closed properly, then create a new zstd file write.

from zstd.

Ah that makes sense. Hmmm, unfortunately usually those use cases are not directly handled by compression algorithm, they are usually implemented either at the file format layer (For examples, usually all databases have some sort of repair mechanism, see levelDB for example: https://godoc.org/github.com/syndtr/goleveldb/leveldb#Recover)
Or you could implement it with control codes and/or repair indexes (examples: CRCs: https://en.wikipedia.org/wiki/Error_detection_and_correction or https://en.wikipedia.org/wiki/Error_correction_code)

With that being said, zstd does implement an optional checksum control that is not surfaced yet with this library: #43. PR are welcomed if you want to implement it!

from zstd.

thank you

from zstd.