CI test run does not reflect real production systems about kati HOT 7 CLOSED

test suite seems to be really unstable. Trying all cases (ruby runtest.rb -c, -c -n, -c -n -a) on Ubuntu 14.04, Fedora 23, Fedora 23 with make 3.81 in $PATH leaves me with 1 to 8 unexpected passes/failures.

I wish I had at least one combination with 0 :-/

from kati.

Now runtest.rb sets SHELL=/bin/bash and "ln -sf /bin/bash /bin/sh" was removed from the travis CI setting: fc24bd2 I also made a bunch of changes for GNU make 4. Hopefully these changes mitigate your issue?

from kati.

Your latest changes improve the situation somewhat. "ruby runtest.rb -c" now passes on my Ubunut 14.04 machine.

But other tests can still fail, I think there is something wrong with handling of echo handling when using ninja?

err_error_in_recipe.mk: FAIL
--- out.make    2016-05-03 09:56:37.074432172 +0300
+++ out.kati    2016-05-03 09:56:37.074432172 +0300
@@ -1,4 +1,5 @@
 === test ===
+/bin/bash -c "echo -e \"Makefile:2: *** foo.\" 2>&1 && false"
 Makefile:2: *** foo.

 === FILES ===
fail_ignore_error.mk: FAIL
--- out.make    2016-05-03 09:56:38.302433053 +0300
+++ out.kati    2016-05-03 09:56:38.302433053 +0300
@@ -1,5 +1,6 @@
 === test ===
 *** [test] Error 1
+/bin/bash -c "(false ) && (false ; true ) && (echo FAIL )"

 === FILES ===

fail_subshell_in_recipe.mk: FAIL
--- out.make    2016-05-03 09:56:38.330433073 +0300
+++ out.kati    2016-05-03 09:56:38.330433073 +0300
@@ -1,5 +1,6 @@
 === test ===
 *** [test] Error 1
+/bin/bash -c "(false ) && ((true) ; echo FAIL )"

 === FILES ===

from kati.

Could you tell me the result of

$ cd out/err_error_in_recipe.mk && ninja

For me, the output is

[1/1] build test
FAILED: /bin/bash -c "echo -e "Makefile:2: *** foo." 2>&1 && false"
Makefile:2: *** foo.
ninja: build stopped: subcommand failed.

The line with "FAILED:" should be removed by https://github.com/google/kati/blob/master/runtest.rb#L114 . But it looks like your ninja doesn't have the FAILED: prefix?

from kati.

I have two lines, where you have one line:

[1/1] build test
FAILED: test
/bin/bash -c "echo -e \"Makefile:2: *** foo.\" 2>&1 && false"
Makefile:2: *** foo.
ninja: build stopped: subcommand failed.

Maybe you are not using upstream master of ninja?

from kati.

Ah, it seems very recent ninja outputs two lines. The above change works?

from kati.

Yes, on Ubuntu 14.04 (make 3.81 in $PATH) and Fedora 23 (make 4.0 in $PATH) I now get passing result for -c, -c -n and -c -n -a

Closing this issue.

from kati.