Comments (13)
@OpenFoam-User @aborruso there's a bit more going on here.
$ cat 1461.sh
cat test.csv
echo
mlr --csv clean-whitespace then put '$e = $d + 5; $t = typeof($d)' test.csv
echo
mlr --csv clean-whitespace test.csv | mlr --csv put '$e = $d + 5; $t = typeof($d)'
$ sh 1461.sh
a, b, c, d
1, 2, 3, 4
5, 6, 7, 8
a,b,c,d,e,t
1,2,3,4,(error),string
5,6,7,8,(error),string
a,b,c,d,e,t
1,2,3,4,9,int
5,6,7,8,13,int
- One issue is that with space after comma, the field
" 4"
is type-inferred as string -- whereas without space after the comma,"4"
is type-inferred as int.- One might argue that this is a design flaw (going all the way back to Miller 1). Maybe Miller should try to strip leading or trailing whitespace before type-inferring.
- The other issue is that the
clean-whitespace
verb takes strings to strings.- This means that with
then
and no|
, the string" 4"
becomes the string"4"
, and adding 5 to a string is an error. - Whereas with the
|
rather than thethen
, the string"4"
is type-inferred as int. - One might argue that
clean-whitespace
, after having done its job, should re-infer types.
- This means that with
I agree that at very least this needs some documentation -- however, I'm not sure that's enough. I'll think on this a bit.
from miller.
Of the two suggestions above, namely:
One might argue that this is a design flaw (going all the way back to Miller 1). Maybe Miller should try to strip leading or trailing whitespace before type-inferring.
One might argue that clean-whitespace, after having done its job, should re-infer types.
I think the first one is perhaps too disruptive. I think that if I were starting afresh, this might be the best option. However, I suspect (without proof or data) such a change might break existing scripts and/or expectations for people.
The second one seems elegant and is implemented in PR #1464.
from miller.
A personal note. I think Miller is really fantastic for these reasons:
- it works great and it has important and useful features, which all together are hardly found in other products;
- it is very well documented;
- @johnkerl is a very good developer and a very generous person
from miller.
Thank you for your reply. Sorry I don't understand your reply. I am not looking for strings. Please see my updated command that works and the expected result.
If you run the command in one shot, without exit from it, for miller those fields that contain spaces are strings.
If you run
mlrgo --csv clean-whitespace then put '$type_b=typeof($b)' c.csv
you get
a,b,c,d,type_b
1,2,3,4,string
5,6,7,8,string
If exit from it and pass the cleaned output via stdout again to mlr, those strings have become numbers and so everything works.
from miller.
If now this issue is ok for you, please close it
Yes, it's clear for me, thanks to your explanation. But I suspect this would be clear for new users. Probably, before closing the issue one should clarify this in the documentation or probably provide a way to force clean-whitespace
to write its output to stdout before the next then
.
from miller.
The second one seems elegant and is implemented in PR #1464.
Perfect! I have tried the PR on the test file above, and it works as expected.
Many thanks
from miller.
In your file, all cells are strings, because you have those spaces.
Then you should force to set all to strings:
mlr --csv -S clean-whitespace c.csv
to get
a,b,c,d
1,2,3,4
5,6,7,8
from miller.
To run your put verb
mlr --csv -S clean-whitespace then put '$f=5;$f+=int($b)' c.csv
from miller.
Hi @aborruso
Thank you for your reply. Sorry I don't understand your reply. I am not looking for strings. Please see my updated command that works and the expected result.
Thank you
from miller.
Thank you for the detailed answer. Now I understand. But this behaviour is not documented in the documentations. As a beginner I spent 5 hours to narrow down the issue to clean-whitespace and put command.
from miller.
I generally, when I work with CSVs, which is a format with so many flaws, the first thing I do is normalize them. So, with a file like that I usually first apply stand-alone clean-whitespace.
I, for example, did not realize that only the "a" field was without spaces and I was making casting errors.
from miller.
If now this issue is ok for you, please close it
from miller.
Yes, it's clear for me, thanks to your explanation. But I suspect this would be clear for new users. Probably, before closing the issue one should clarify this in the documentation or probably provide a way to force
clean-whitespace
to write its output to stdout before the nextthen
.
You could open a PR to propose a change in the documentation.
from miller.
Related Issues (20)
- Bash process substitution not working with `put -f`
- Miller's `strptime` accepts fewer format options than `strptime`
- Inconsistent result when using `$*`
- Double-width characters spoil column alignment HOT 4
- `mlr --icsv --ojson cat < mlr.bug.csv` drops some columns HOT 5
- Add description for "put" verb HOT 1
- 'mlr cut' is very slow HOT 8
- mlr --otsv does not handle broken quotes correctly compared to --ocsv HOT 6
- JSON to CSV Error HOT 8
- exit code = 1 for --csv skip-trivial-records and csv file's last record is blank
- Automated way of clearing down column data HOT 7
- JSON flag documentation question HOT 1
- Equivalent to Excel function "data load from folder" then "combine and load" multiple CSV's finally "apply transformations" HOT 7
- Find and remove "string" retaining all other row data HOT 5
- Find and replace special character & with and using ssub HOT 2
- Add a `stat` DSL function HOT 4
- Notwithstanding --skip-comments, double-quoted tokens in comment are mishandled with any --c2* option HOT 3
- Completely unexpted output of put, then cut, then label HOT 3
- do we have python binding? HOT 1
- About `--transpose` in the documentation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from miller.