Prepipe and zipped HTTP source about miller HOT 6 CLOSED

@aborruso that very much surprises me -- there is something I don't understand here & will need to dig into ...

from miller.

Thank you

from miller.

If I change the source, it works

mlr --gzin --t2c head "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/BOP_EU6_M?format=TSV&compressed=true"

from miller.

@aborruso I believe it's because .zip and GZIP are different things

from miller.

@aborruso I believe it's because .zip and GZIP are different things

Ok, but for the same source, the same zip file, I can read it locally

mlr --prepipe gunzip --csv head eurofxref-hist.zip

but I have error if I read it from its URL

mlr --prepipe gunzip --csv head "https://www.ecb.europa.eu/stats/eurofxref/eurofxref-hist.zip"

Shouldn't it still read it, both locally and remotely, just like with gzip?

from miller.

@aborruso sorry for the long delay.

This was (in hindsight) obvious to me.

• Most programs like wc and cat and gunzip et al. operate on local files -- it would be very surprising for any of them to accept https://... URLs as input. For such tools you need to do two steps: (a) copy to local, e.g. with wget, and (b) use wc, cat, gunzip, etc on the local copy.
• Miller tries to go "above and beyond" by allowing you to read from https://... URLs

The key insight is that when you do

mlr --gzin --t2c head "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/BOP_EU6_M?format=TSV&compressed=true"

then you are asking Miller -- which goes "above and beyond" by accepting https://... URLs as input -- to open that URL, and run the data it reads from the internet, and run that through its internal zip-decoder library. This works.

But when you do

mlr --prepipe gunzip --csv head "https://www.ecb.europa.eu/stats/eurofxref/eurofxref-hist.zip"

then you are asking Miller to do (effectively)

gunzip https://www.ecb.europa.eu/stats/eurofxref/eurofxref-hist.zip | mlr --csv head

and of course the gunzip program does not go "above and beyond" by accepting https://... input.

So there is (unfortunately) nothing I can do about this.

A side note: why did I even invent mlr --prepipe? It's because if you have

mlr --prepipe foo --... a.csv b.csv c.csv

that's simpler than having to do

for file in a.csv b.csv c.csv; do
  foo $file | mlr --...
done

from miller.