make failing about miller HOT 54 CLOSED

Looks like miller switched to autoconf in #67. Try this:

autoreconf -if
./configure
make

from miller.

there's also this:
https://travis-ci.org/johnkerl/miller/jobs/81385366

 % autoreconf -if
autoreconf2.50: 'configure.ac' or 'configure.in' is required

from miller.

Looks like that's fixed in fc3db29.

from miller.

And does autoreconf -if work for you?

from miller.

Yup, using autoconf v2.69 and automake v1.15 on Linux. If yours is still broken with current master, what error are you getting?

from miller.

Hello,

% autoconf --version
Autoconf version 2.13

% automake --version
automake (GNU automake) 1.14.1

% cd ~/bin/miller 

% git pull
Already up-to-date.

% cd c

% autoreconf -if
autoreconf2.50: 'configure.ac' or 'configure.in' is required

You're on a very newer version of autoconf than I am, so that's something worth looking into.

However....

FreeBSD  11.0-CURRENT FreeBSD 11.0-CURRENT #0 r287930: Thu Sep 17 20:40:12 UTC 2015

% autoconf --version
autoconf (GNU Autoconf) 2.69

% automake --version
automake (GNU automake) 1.15

% pwd
/home/vagrant/bin/miller/c

% autoreconf -if
autoreconf-2.69: 'configure.ac' or 'configure.in' is required

Are you sure autoreconf -ifis the right command?

from miller.

Ah right, sorry, run autoreconf -if from the repository root, not c/. configure.ac should be in that directory.

from miller.

ugly:

https://gist.github.com/jungle-boogie/8da8bf72194f6d42a07d

from miller.

That looks like you don't have libtool installed; try installing that and trying again?

from miller.

Not really any better...
https://gist.github.com/jungle-boogie/49b38c8daac24980586d

https://gist.github.com/jungle-boogie/9c0a20ae0d0cfd0a40e1

make -e CC=clang in millier and c directory immediately fail:
https://gist.github.com/jungle-boogie/cc46e900ff75dc73dd83

make generates many of the warnings that previously existed and eventually fails with this:
https://gist.github.com/jungle-boogie/62544a083794897ccc92

I know autoconf and automake can do wonderful things but now the dependencies are increased, the warnings are not corrected and portability seems smashed to pieces.

from miller.

just fyi, this build installs:
https://github.com/johnkerl/miller/tree/v2.2.0

This is Miller version >= v2.2.0.

so to re-cap, people will now need to have:

autoconf
libtool
automake
and basically gcc compiler and/or gnu/linux.

from miller.

In an autotools workflow you'd usually specify the C compiler at the configure stage, something like CC=clang ./configure && make. (I don't have a clang on my current machine and haven't tested this.)

from miller.

If you take a look at this build output: https://travis-ci.org/johnkerl/miller/jobs/81531717
clang is set in the environment and if I do that, followed by make -f Makefile.no-autoconfig -e yes, it will work.

so technically, it will build but it's much more cumbersome.

from miller.

Just tested with clang; from a fresh checkout, autoreconf -if && CC=clang ./configure && make works for me without jumping through any additional hoops.

from miller.

My apologies. This is a bit in flux with current status on #9.

The short of it:

• cd /path/to/miller/c; make -f Makefile.no-autoconfig is the same build as always. No new dependencies.
• cd /path/to/miller; autoreconf -fiv; ./configure; make is the way to build from master.
• If you've got a release tarball (the next will be mlr-2.3.0.tar.gz although I haven't released that yet) then I hope you'll be able to get away with ./configure && make with no need for autotools.

The long of it:

• I'm a veteran of the software industry in many ways but am brand-new to internals of autoconfig.
• The fine work done by @0-wiz-0 is a fantastic gift to Miller, which will greatly enhance its portability.
• At this point my most important work (detailed on #9) is to clearly and cleanly iron out what the dependencies are for master builds and for tarball builds, and to document everything for the benefit of myself and others.

from miller.

Also @jungle-boogie, I do agree with you about autoconf -- I do (personally) find its error-handling confusing. Yet, many folks find it familiar and essential -- and in fact will be reluctant to use any open-source software which doesn't have autoconf support. These are two different use-cases and they both need supporting.

For those reasons:

• I'll make autoconf the default (and will connect it into the Travis build), since when it works, it works on all sorts of platforms for all sorts of people;
• I'll keep Makefile.no-autoconfig around, since it's little more than gcc -I. *.c */*.c, it's easier to figure out in cases of confusion, it builds quite quickly and transparently, and it's adequate on various Linuxes and OSX
• I'll document both alternatives
• Most importantly I'll include up front what the dependencies are for various use-cases

from miller.

Lastly, I don't like to break head -- but effectively I did with last night's push (I woke up to find this conversation), by failing to communicate then what I just wrote now. This is my bad.

from miller.

Fourthly, @olorin is correct that you can pick your C compiler with options to configure -- likewise with install-dir and so on. So part of the miller+autoconf doc effort will be to tabulate for easy reference some of the most-used autoconf options. (The automake.pdf manual is 100+ pages; as a service to users I certainly should distill/curate a short list of most-needed information out of that.)

from miller.

Hi @olorin and @johnkerl,

Breaking head is better than regressions in a release so I'm happy to help find all these things before the next release. ;)

I'd give the slight recommendation that documentation get updated (when it's known to be wrong) when commits like this are made.

With what was said above, I'm still getting a build failure with
autoreconf -if && CC=clang ./configure && make:
https://gist.github.com/jungle-boogie/ffce6ee4b3a0ed3da244

autoreconf -fiv; ./configure; make also fails:
https://gist.github.com/jungle-boogie/9bf651cc084958dea36c

I project I use that works with automake and autoconf is tmux:
https://github.com/tmux/tmux

When building tmux, I don't need to specify the compiler or anything outside of the few steps on the readme once I have the dependencies installed. Will it ALWAYS be necessary to specify the compiler when it's not gcc?

from miller.

The error output looks like you built your libtool for gcc, not clang.
Make sure you use the same compiler for both, you can't switch.

For configure, it's better to give the compiler on the command line instead of the environment, i.e.
./configure CC=clang
instead of
CC=clang ./configure

then configure remembers it when rebootstrapping due to changed autotools files.

from miller.

@jungle-boogie, agreed about doc update lagging build-change -- when I said "this was my bad" above, that's what i meant.

from miller.

Hi @0-wiz-0,

so do I simply do ./configure CC=clang or autoreconf -if && ./configure CC=clang && make

Output of first:
https://gist.github.com/jungle-boogie/70eb3f87588750e9e228

Output of second:
https://gist.github.com/jungle-boogie/1c5c4056308fde4c87b0

these were both performed in miller directory, not the miller/c directory.

@johnkerl,

Got it. thank you for your project!

from miller.

Please make sure, as I already said above, that libtool was compiled for clang. Or just use default cc for compiling miller (i.e., remove CC=clang).

from miller.

Hi @0-wiz-0,

Yes, I did see you mentioned that and I'll check into how libtool was compiled.

By saying just use default cc, you mean gcc?

from miller.

Whatever 'cc' is. It seems you're running FreeBSD 10, so it might even be clang.
"cc --version" will probably tell you.

from miller.

Yes, I'm testing this on freebsd and it is clang:

cc --version
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd10.2
Thread model: posix

I'm pretty sure the libtool package for freebsd was built with clang but I'm checking.

from miller.

Hi @0-wiz-0,

Just confirmed that libtool package is built with clang.

Thanks!

from miller.

And just as a last datapoint, what does:

make distclean
autoreconf -fiv
./configure
make check

report?

I.e. blow away previous build, run autoconf and configure again (without CC) and make?

from miller.

Here's the report:
https://gist.github.com/jungle-boogie/2d2186c27660932b2abf

I first did a git pull followed by make clean and then the commands above. make check is what's reported.

from miller.

That's a different problem now. A circular dependency was introduced by
fc3db29

The cli library currently depends on the container library which contains on the mapping library, which depends on the cli library. John, can you please clean this up?

from miller.

sure

from miller.

It's still unclear to me how this got past Travis ...

from miller.

Does travis start from a clean checkout every time?
If the libraries previously existed from an earlier build, it probably works.

from miller.

Here's the most recent failure log:
https://gist.github.com/jungle-boogie/832e7a5a56703bc980b8

from miller.

Yes, Travis is starting from git clone. Unless it is retaining artifacts in some way I don't understand.

I'm fixing this, but there is a deeper issue with cyclic subpackage dependencies involving the join function. @jungle-boogie you might avoid master until I resolve this issue.

from miller.

Hi @johnkerl,

Thanks for the tip and your efforts to correct this!

from miller.

meanwhile, #dayjob ;)

from miller.

The last error jungle-boogie reported is quite similar to the first one.
My current best guess is that there is a problem with FreeBSD's clang and -pg.
Can you try building "mlrp" using the old Makefile? It is not built by default in the pre-autoconf Makefile, so perhaps this issue existed before and just became more visible up now.

from miller.

Hi @0-wiz-0,

Is that something I can do or does it need to be done on master?

from miller.

cd /path/to/miller/c; make -f Makefile.no-autoconfig mlrp should do it.

from miller.

% cd c; make -f Makefile.no-autoconfig mlrp
make -C dsls -f Makefile.no-autoconfig put_dsl_parse.o
gcc -o lemon lemon.c
make[1]: exec(gcc) failed (No such file or directory)
*** Error code 1

Stop.
make[1]: stopped in /usr/home/sean/bin/miller/c/dsls
*** Error code 1

Stop.
make: stopped in /usr/home/sean/bin/miller/c

from miller.

Well, of course, please also add the CC=clang part you always have. So.
make -f Makefile.no-autoconfig CC=clang mlrp
Thanks.

from miller.

Hello,

https://gist.github.com/jungle-boogie/cc7e50db46084af2df81

from miller.

Ok, so it's a problem with the mlrp executable and independent of the build system. One step forward.
Thanks for the test!

from miller.

@0-wiz-0,

Thank YOU for walking me though all of this.

from miller.

I believe the originally reported problem is resolved by my latest commit (51f89ef), although #9 is still open re html/doc info.

Re the second problem, mlrp -- I don't doubt that BSD semantics for gprof are different. @jungle-boogie I appreciate your help (if you really want to run the profiled version on your system) since I don't have access to a BSD system. (Well, OSX ;)) Can you try what the error message suggested, namely, after running your commands, follow that with simply pasting in at the command prompt

clang -g -std=gnu99 -I. -I.. -Wall -Werror -g -pg *.c cli/*.c lib/*.c containers/*.c stream/*.c input/*.c mapping/*.c output/*.c ./dsls/put_dsl_parse.o    ./dsls/put_dsl_lexer.o    ./dsls/put_dsl_wrapper.o ./dsls/filter_dsl_parse.o ./dsls/filter_dsl_lexer.o ./dsls/filter_dsl_wrapper.o -lc -lm -o mlrp

? If so that'll indicate that profiling on BSD needs -lc on the link line.

from miller.

@johnkerl,

Will I run that in c/ or millier/?

from miller.

After running make -f Makefile.no-autoconfig CC=clang mlrp I ran what's above and the output:

/tmp/mlr_test_util-162c18.o: In function `write_temp_file_or_die':
/home/sean/bin/miller/c/lib/mlr_test_util.c:9: warning: warning: mktemp() possibly used unsafely; consider using mkstemp()

from miller.

OK so it looks like there were no errors, so if you ls -l mlrp & see it's as old as the command you typed, then that suggests that it linked OK.

So two things: one, for me, is the data point that to use gprof on FreeBSD, one needs to link with -lc. The other, for you, is if it successfully created the mlrp executable, you can go ahead and do whatever you intended to do with a gprof-enabled Miller executable.

from miller.

Yes, mlrp has the same timestamp as when I ran the command you provided above.

from miller.

Great, you're good to go then! I'm happy to hear whatever you find out about profiling Miller on FreeBSD.

from miller.

mlrp aside, should this occur with sudo make -f Makefile.no-autoconfig CC=clang mlr install: ?

https://gist.github.com/jungle-boogie/c87f1ca07be45b4adc26

from miller.

My bad. Resolving the circular-dependency issue you found, but didn't compile with/without autoconfig before pushing.

670c5c5
711c300

from miller.

👍

from miller.