Allow stdout redirect for further processing about angle-grinder HOT 14 CLOSED

from angle-grinder.

from angle-grinder.

I was running:

cat loggers.txt| agrind '* | parse "*" as logger | count by logger' > foo

where loggers.txt just contained a simple string.

I first ran the bash command without the > foo to time it. It took 80 seconds and output as expected.

I then ran it with the redirect, waited more than 2 minutes and then CTRL-C'd the process. but foo is empty still.

$ agrind --version
ag 0.7.2

(brew installed).

Example file:

$ head loggers.txt
[OrbStatisticsControl
[it.timing.LoggerSink
[essFaxDeliveryAction
[it.timing.LoggerSink
[it.timing.LoggerSink
[main.TasksRepository
[it.timing.LoggerSink
[it.timing.LoggerSink
[it.timing.LoggerSink
[it.timing.LoggerSink

from angle-grinder.

from angle-grinder.

ah yes, using head (to make the file input smaller) works fine:

psmith@paul ~/temp/au1 $ head loggers.txt| agrind '* | parse "*" as logger | count by logger' > foo
psmith@paul ~/temp/au1 $ cat foo
logger                       _count
-------------------------------------------
[it.timing.LoggerSink        7
[main.TasksRepository        1
[OrbStatisticsControl        1
[essFaxDeliveryAction        1

So input/output buffering issue?

I'd be happy to try to build from source and try out the tip of branches (good chance to get my shirt sleeves rolled up with the Rust ecosystem)

from angle-grinder.

from angle-grinder.

Running 0.7.4 via git checkout and cargo install. Same behaviour.

interesting observations:

• OSX shows agrind buring about 1 core, so it's definitely doing something
• Spindump (attached) shows the threads doing merge-sort so that does also indicate it's "doing something"

So your idea on performance difference based on the standard output does sound about right, but I am not sure why that would be so. Will be interesting to see what you find!

agrind-Spindump.txt

from angle-grinder.

Also for context, the size of this file is 757MB:

$ wc  loggers.txt
 36071659 36073508 793576498 loggers.txt

from angle-grinder.

from angle-grinder.

Should be fixed by 7eae04e

from angle-grinder.

By the way @tallpsmith not sure what kind of cumulative you mean, but you can do things like:

* | parse "*" as logger | count by logger | sum not sure if that's what you meant

from angle-grinder.

taking the tip of master (which still says 0.7.4 btw, even though there's a 0.7.5 tag)

psmith@paul ~/temp/au1 $ time cat loggers.txt| agrind '* | parse "*" as logger | count by logger' > foo

real	1m15.623s
user	2m15.256s
sys	0m10.300s

Great Success! https://media.giphy.com/media/a0h7sAqON67nO/giphy.gif

(and it's even faster than with it displaying to tty..)

from angle-grinder.

Also in response to:

* | parse "*" as logger | count by logger | sum not sure if that's what you meant

 cat loggers.txt| agrind '* | parse "*" as logger | count by logger | sum'
Failure parsing query. Parse failed at: Error(Code(CompleteStr("| sum"), Eof))

I'm not sure how the heck Sumologic does it, but I was thinking of something like the accum operator: https://help.sumologic.com/Search/Search-Query-Language/Search-Operators/accum

So a running total along side the previous column. That was the idea I was getting towards anyway.

from angle-grinder.

thanks for the quick turn around!

from angle-grinder.