google / crush-tools Goto Github PK

What steps will reproduce the problem?
1. make; make check

What is the expected output? What do you see instead?
The script reports:
PASS: reutils: make flags.
PASS: reutils: resubst compile.
PASS: reutils: basic substitution.
PASS: reutils: global substitution.
PASS: reutils: variable substitution.
PASS: reutils: multiple variable substitution.
FAIL: reutils: malformed variable substitution 3: expected "he${1o", got 
"hellhe${1oo".
FAIL: test/reutils_test

So, it looks like the re is hanging onto previous results. Also, the 
failing test is mis-numbered as '1' when it is really '3', but that isn't 
particularily at issue.

Workaround:
The specific test case that fails is a test-bad-data test, and can be 
safely commented out as it indicates that the re engine is broken in the 
case when the re is broken.

Config Details:
[jrenwick@machine (CRUSH) 3975]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
[jrenwick@machine (CRUSH) 3975]$ uname -a
Linux machine 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 
x86_64 x86_64 GNU/Linux

It would be hot if aggregate and aggregate2 supported min/max calculations.

The current need looks something like:
- I have a weblog that I am appending to over time with each day's data.
- I only care about the last 30 days data.
- But - I need to know some event and every time it happens, the growing
weblog file updates that event's time to the current date so as to keep it
current in the 30 day window.

I can picture something like:

cat huge_weblog | aggregate2 -F Date-Field,Data-Field -M Date-Field -S
Data-Field-Counter | drop_older_than_30_day_data > huge_weblog

Where input data looks something like:

Date-Field,Data-Field,Data-Field-Counter
03-19-2009,1,3
03-20-2009,1,1

And output data looks something like:

Date-Field,Data-Field,Data-Field-Counter
03-20-2009,1,4

Add an option in configure.ac so the default delimiter can be specified at 
build time.  This will allow the tools to be tailored to the user 
environment better.

Some of the apps written in Perl have dependencies on other modules that 
may not be present.  configure.ac should check for these and either a) 
fail; or b) conditionally skip building/installing scripts whose 
dependencies are not met.

There are instances where malformed input causes some apps to crash.  E.g., 
when an incorrect delimiter was specified or a line of input doesn't 
contain as many fields as expected.  At the very least, the apps should 
`exit(EXIT_FAILURE)` instead of seg-faulting.

make  all-am
gcc -DHAVE_CONFIG_H -I. -I../../src/libcrush     -g -O2 -MT filterkeys.o
-MD -MP -MF .deps/filterkeys.Tpo -c -o filterkeys.o filterkeys.c
mv -f .deps/filterkeys.Tpo .deps/filterkeys.Po
gcc -DHAVE_CONFIG_H -I. -I../../src/libcrush     -g -O2 -MT main.o -MD -MP
-MF .deps/main.Tpo -c -o main.o main.c
mv -f .deps/main.Tpo .deps/main.Po
gcc -DHAVE_CONFIG_H -I. -I../../src/libcrush     -g -O2 -MT usage.o -MD -MP
-MF .deps/usage.Tpo -c -o usage.o usage.c
mv -f .deps/usage.Tpo .deps/usage.Po
/bin/sh ../../libtool --tag=CC   --mode=link gcc  -g -O2   -o filterkeys
filterkeys.o main.o usage.o ../libcrush/libcrush.la 
mkdir .libs
gcc -g -O2 -o .libs/filterkeys filterkeys.o main.o usage.o 
../libcrush/.libs/libcrush.dylib 
Undefined symbols:
  "_open64", referenced from:
      _filterkeys in filterkeys.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[3]: *** [filterkeys] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Thanks.

reorder silently truncates fields > 511 bytes.  I'm using crush-tools on apache 
log output, and it's quite easy to have fields significantly longer that.  Long 
term, maybe it should be a cmd line option and malloc'ed (once).  Short term, 
making it something much larger (16k?) doesn't seem like it would significantly 
affect performance or memory use (it's a single static buffer, AFAICT)

What steps will reproduce the problem?
1. configure && make

What is the expected output?
no errors

What do you see instead?

[...]
Making all in cutfield
make  all-am
gcc -DHAVE_CONFIG_H -I. -I../../src/include     -g -O2 -MT cutfield.o -MD
-MP -MF .deps/cutfield.Tpo -c -o cutfield.o cutfield.c
cutfield.c: In function ‘cutfield’:
cutfield.c:73: error: invalid lvalue in assignment
cutfield.c:118: error: invalid lvalue in assignment
make[3]: *** [cutfield.o] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1


What version of the product are you using?
crush-2008-10.tar.gz

On what operating system?
Mac OS X 10.4.11

Please provide any additional information below.
The problem seems to be the reassignment to stdin (cutfield.c:73):

stdin = nextfile(argc, argv, &optind, "r");

Using a local variable should fix the problem.

What steps will reproduce the problem?
1. Search for gzdog in examples:  
http://code.google.com/p/crush-tools/source/search?q=gzdog&origq=gzdog&btnG=Sear
ch+Trunk
2. Note examples which refer to gzdog, swap with zcat

What is the expected output? What do you see instead?
Note examples which refer to gzdog, swap with zcat

What version of the product are you using? On what operating system?
n/a

Please provide any additional information below.

Enhance pivot to support text-only pivot mode.

Currently, value fields can only be pivoted to numbers - consider allowing 
pivots for text values as well.

See - e.g.  

http://groups.google.com/group/crush-tools/browse_thread/thread/35c9e561e84b1b57

Review filterkeys for inclusion into branch


After the review, I'll merge this branch into:
/trunk

During `make check', reutils_test should be skipped when pcre is not 
present.

It is also a good idea to add a warning at the end of ./configure when pcre 
is not detected.

What steps will reproduce the problem?
1. ./configure
2. make
3. make check

What is the expected output? What do you see instead?
Expected all tests to pass but found one failing test:
/bin/sh: line 1: 23971 Segmentation fault: 11  ${dir}$tst
FAIL: test/ffutils_test

What version of the product are you using? On what operating system?
Version 2010-03 on 64-bit Mac O X Lion on Intel 64-bit processor.

Please provide any additional information below.
The test output is attached as files.

What steps will reproduce the problem?

  $ echo -n '' | funiq 
  Segmentation fault

  Or on another system:

  $ echo -n '' | funiq
  funiq: xmalloc failed to allocate 1125870855412416 bytes: Cannot
  allocate memory


What is the expected output? What do you see instead?

  0-byte output.

What version of the product are you using? On what operating system?

  trunk, r504

What steps will reproduce the problem?

$cutfield -d , -F my_label
other_label,yet_another
Segmentation fault
$

There's a bug in cutfield that segfaults when the field label specified by -F 
is not present in the header.
Basically expand_label_list from ffutils.c returns a negative ssize_t . the 
call in cutfield.c:79 stores the result in a size_t. When expand_label_list 
returns a negative result, cutfield sees it as a large positive. 
Attached is the one-line diff for cutfield.c and a test case for this scenario. 

What steps will reproduce the problem?
1. ./configure
2. make
3. make check

What is the expected output? What do you see instead?
I expect al the tests to pass but three of them fail.

What version of the product are you using? On what operating system?
crush-tools-2010-03.tar - Mac OS X 10.6.8 (10K549)

Please provide any additional information below.

What steps will reproduce the problem?
1. Compile latest stable rev of crush-tools on FreeBSD 8.2-RELEASE 
2. Run following script https://gist.github.com/1362440

What is the expected output? What do you see instead?

Expected output is -- converted logfiles in specified folder (as one can see 
from script i linked), but I only get segmentation fault:

./squid_log_aggregate.sh: line 16: 35417 Exit 141                ( zcat 
$LOGDIR/access.log.2.gz && cat $LOGDIR/access.log.1 )
     35418 Segmentation fault: 11  (core dumped) | convdate -i "%s" -o "%F %H:%M:%S" -F time -p

What version of the product are you using? On what operating system?

I am using 2010-03-Release version of crush-tools, compiled on FreeBSD 
8.2-RELEASE.

Please provide any additional information below.

Please, notice, that FreeBSD m4 does not support the '--prefix-builtins' 
optiion, so I used gm4 (GNU-m4) while compiling crush-tools.

One feature that could be useful:

fieldsplit could have an option to split your data into a fixed number N 
output files based on a hash of the value for a field so that all records 
for a particular value will end up in the same bin.  Even if there are 
many values of field "foo", you can still break up your problem into 
manageable chunks.

What steps will reproduce the problem?
1. ./configure
2. make

What is the expected output? What do you see instead?

expecting make to complete, instead I get this:

/usr/bin/m4 -D"CRUSH_PERL_PATH=/usr/bin/perl" --prefix-builtins 
-I./../perllib hackery.m4  crush_version.pl > fieldsplit
m4: illegal option -- -
usage: m4 [-d flags] [-t name] [-gs] [-D name[=value]]...
          [-U name]... [-I dirname]... file...
*** Error code 1


What version of the product are you using? On what operating system?

crush-2009-01
FreeBSD 7.0-RELEASE

Please provide any additional information below.

the version of m4 installed does not appear to support the
--prefix-builtins option

[cglass@oddjob ~/crush-2009-01]$ pkg_info | grep m4
m4-1.4.13,1         GNU m4

I am not sure if this is a general problem but I have noticed that the
--prefix-builtins option is missing in FreeBSD 8.0-RELEASE also.

I have had no problems building crush on linux.

Any help would be appreciated especially if this is a known problem, please
excuse me if I have made an obvious mistake.

Thanks

Charles

What steps will reproduce the problem?

[(xfp-custom/goog) google3]$ echo -e "1,X\n2,Y" > /tmp/x
[(xfp-custom/goog) google3]$ cat /dev/null | hashjoin -f /tmp/x -k 1 -l 1 -j 2
hashjoin: xmalloc failed to allocate -1 bytes: Cannot allocate memory

What is the expected output? What do you see instead?

A nicer error message that stdin (or the joined file) is 0 bytes.

What version of the product are you using? On what operating system?

(xfp-custom/goog) google3]$ hashjoin -V
0.1

Please provide any additional information below.