Similar test failures in multiple languages about mal HOT 25 CLOSED

@vi Yes, I'm pretty sure this is a result of readline behaving in a way that runtest.py is not expecting. What OS and version are you running? Also, do you know what version of readline you are using on your system? Also, do any other languages pass or have you just tried rust, python and perl?

from mal.

Debian wheezy/sid. How do I look for readline version? Maybe it's 6.2+dfsg-0.1 0. Some other languages (like Go or Forth or C). CSharp passes this test, but still have 1 failure in a some readline test. Make and bash versions fail the same way.

Why should the tests be readline-enabled by default? I except them to use clear mode (except of the tests dealing with readline explicitly).

Maybe the "Step 0" should be modified to include a readline-less mode? Readline niceties look like an optional feature, there can be just "read bytes until \n" at this stage. Scripting through the readline seems like a hack.

from mal.

@vi some of the implementations support a --raw flag to disable avoid readline/editline support. It is an optional feature, but most of the implementations do have it and it's a really nice feature to have if it's available. The tests run across all the implementations correctly for me so I'm not sure what's happening in your environment. You can try using runtest-old.py. It uses pexpect and it's about an order of magnitude slower, but it might work more reliably for you.

from mal.

@vi I also just added a --no-pty option to runtest.py This should run the the implementations without using a tty/pseudo-tty (i.e. just using pipes). However, this mostly doesn't help for me. The behavior of the various readline/editline/linenoise implementations seems to be all over the map when not run in a tty/pty. The test uses display of prompt and echo to correlate tests to output and to determine if the test has timed out. So I really need to figure out some way to get readline to show the prompt and echo input without sending control codes (i.e. assuming cooked terminal). Ideas?

from mal.

The --no-pty option does not change the behaviour:

$ ./runtest.py --no-pty  q.mal -- rust/target/release/step1_read_print 
TEST: (+ 1 2) -> ['','(+ 1 2)'] -> FAIL (line 2):
    Expected : '(+ 1 2)\r\n(+ 1 2)'
    Got      : '(+ 1 2)\x08\x08\x08\x08\x08\x08\x08\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\r\n(+ 1 2)'
FAILURES: 1

I think it's better to require --raw (or analogue) across implementations to be testable (maybe making it default). Personally I don't typically link to readline in my programs at all, as rlwrap is sufficient for enhancing usability in such simple cases.

In short, MAL seems to love readline too much. They should be in cooler relationship.

from mal.

I'm not a fan of rlwrap compared to having REPLs just DTRT. Rlwrap is great idea, but it never quite works right for me out of the box and it's far less likely to be installed on random systems compared to readline or editlib. If --no-pty is still giving you that same behavior then there is something seriously broken about readline's behavior on your system. Your readline is failing to detect that the stdin/stdout is non-interactive and is still sending ANSI/VT control codes when it shouldn't. The brokenness that I expect to see with --no-pty is that the test just times out because readline decides not to write a prompt or to echo input. Sending control codes to a stdout of a non-interactive terminal is just all wrong.

BTW, I just fired up Debian 7 wheezy in docker, installed python and perl and was able to run the python and perl tests just fine. And this is with libreadline6:amd64 6.2+dfsg-0.1 0.

Here is the Dockerfile if you want to try yourself:

FROM debian:wheezy

RUN apt-get -y update
RUN apt-get -y install make python perl

WORKDIR /root/mal

Then from your mal checkout:

docker build -t mal-test-debian-wheezy .
docker run -i -t --volume=`pwd`:/root/mal mal-test-debian-wheezy ./runtest.py tests/step1_read_print.mal -- python python/step1_read_print.py

from mal.

Tried with NixOS, it works better (no extra terminal codes at least).

But both runtest.py's --no-pty and perl version's --raw hangs and times out there.

from mal.

I encountered the similar problem when I'm trying to write one for GNU Guile. In the beginning I feel very strange why all the implementations added readline feature. I didn't think more and just added it.
But now I can't pass after I replaced the old reader with realine featured one:

Running: ../runtest.py  ../tests/step0_repl.mal -- guile ../guile/step0_repl.scm 
TEST: hello world -> ['','hello world'] -> SUCCESS
TEST: abcABC123 -> ['','abcABC123'] -> SUCCESS
TEST: ;:() []{}"'* -> ['',';:() []{}"\'*'] -> FAIL (line 8):
    Expected : ';:() []{}"\'*\r\n;:() []{}"\'*'
    Got      : ';:()\x08\x08\x1b[C\x1b[C []\x08\x08\x1b[C\x1b[C{}\x08\x08\x1b[C\x1b[C"\'*\r\n;:() []{}"\'*'
FAILURES: 1

My OS is openSUSE-12.2, and readline is libreadline6. This couldn't be the problem of Guile IMO.

from mal.

Anyway, it's fine for me to disable readline reader and finish the rest work.

from mal.

@NalaGinrut yes, readline is definitely optional. I personally find it very convenient to implement early because it's streamlines development (I can just up arrow, enter, up arrow, enter to test the current form I'm working on), but it's certainly not necessary at all. Just a nice to have.

@vi so I might have figured out one of the readline issues and the reason things behave differently for you and for me. I think it may boil down to the value of TERM. for the subprocess (which by default is inherited from the parent's environmnet). Readline/line edit libraries behave differently depending on the setting of TERM. I just pushed a change to runtest to force TERM=dumb for the subprocess. The only implementations which still require --raw mode now are C# and VB (mono based) and I suspect it will fix the issues you were seeing (at least the ones where you are seeing VT/ANSI control codes in the output).

@vi I would be interested if this improves the earlier failures you saw. @NalaGinrut if you still have your readline based implementation I think the TERM setting in runtest might fix your problem too and I would be interested to know that.

from mal.

Isn't "up arrow, enter, up arrow, enter" handled well by rlwrap?

from mal.

Something is broken with testing in master, it works worse than before (i.e. fails earlier than in 1380d8c) :

v@l:18:05:39:~/src/git/mal$ make test^rust
----------------------------------------------
Testing test^rust^step0, step file: rust/target/release/step0_repl, test file: tests/step0_repl.mal
Running: ../runtest.py  ../tests/step0_repl.mal -- ../rust/target/release/step0_repl 
TEST: hello world -> ['','hello world'] -> SUCCESS
TEST: abcABC123 -> ['','abcABC123'] -> SUCCESS
TEST: ;:() []{}"'* -> ['',';:() []{}"\'*'] -> FAIL (line 8):
    Expected : ';:() []{}"\'*\r\n;:() []{}"\'*'
    Got      : ';:()\x08\x08\ruser> ;:() []\x08\x08\ruser> ;:() []{}\x08\x08\ruser> ;:() []{}"\'*\r\n;:() []{}"\'*'
FAILURES: 1
make: *** [test^rust^step0] Error 2
v@l:18:05:47:~/src/git/mal$ make test^perl
----------------------------------------------
Testing test^perl^step0, step file: perl/step0_repl.pl, test file: tests/step0_repl.mal
Running: ../runtest.py  ../tests/step0_repl.mal -- perl ../perl/step0_repl.pl 
TEST: hello world -> ['','hello world'] -> SUCCESS
TEST: abcABC123 -> ['','abcABC123'] -> SUCCESS
TEST: ;:() []{}"'* -> ['',';:() []{}"\'*'] -> FAIL (line 8):
    Expected : ';:() []{}"\'*\r\n;:() []{}"\'*'
    Got      : ';:()\x08\x08\ruser> ;:() []\x08\x08\ruser> ;:() []{}\x08\x08\ruser> ;:() []{}"\'*\r\n;:() []{}"\'*'
FAILURES: 1
make: *** [test^perl^step0] Error 2

If I comment out env['TERM'] = 'dumb' the result it the same.

from mal.

@kanaka Indeed, it's fixed.

make test^guile^step0 TERM=dumb
----------------------------------------------
Testing test^guile^step0, step file: guile/step0_repl.scm, test file: tests/step0_repl.mal
Running: ../runtest.py  ../tests/step0_repl.mal -- guile ../guile/step0_repl.scm 
TEST: hello world -> ['','hello world'] -> SUCCESS
TEST: abcABC123 -> ['','abcABC123'] -> SUCCESS
TEST: ;:() []{}"'* -> ['',';:() []{}"\'*'] -> SUCCESS

Thanks!

from mal.

@NalaGinrut okay, good to know. If you merge with master you shouldn't have to specify it to make (runtest should do that for you).

@vi sigh. I think it's failing earlier because I added step0 tests since then (which should flush out some of these issues early for one thing). It still looks to me like readline is behaving inappropriately to me in your environment, but I couldn't tell you why. I'm fairly confident that setting TERM=dumb is the right thing for the subprocess, but I'm not sure why it seems to be ignored for you.

from mal.

@vi can you try again? I'm pushed a change to force the size of the pty for the subprocess to something large (200x100). It's possible readline on your system was assuming something smaller than the normal default (80x25) and so that may have been causing control characters to be added to the stream. I doubt it will fix it, but fingers crossed :-)

from mal.

Tried again. It fails with different output:

v@l:22:50:08:~/src/git/mal$ make test^perl 
----------------------------------------------
Testing test^perl^step0, step file: perl/step0_repl.pl, test file: tests/step0_repl.mal
Running: ../runtest.py  ../tests/step0_repl.mal -- perl ../perl/step0_repl.pl 
TEST: hello world -> ['','hello world'] -> SUCCESS
TEST: abcABC123 -> ['','abcABC123'] -> SUCCESS
TEST: ;:() []{}"'* -> ['',';:() []{}"\'*'] -> FAIL (line 8):
    Expected : ';:() []{}"\'*\r\n;:() []{}"\'*'
    Got      : ';:()\x08\x08\ruser> ;:() []\x08\x08\ruser> ;:() []{}\x08\x08\ruser> ;:() []{}"\'*\r\n;:() []{}"\'*'
TEST: hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{}"'*) -> ['','hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'] -> FAIL (line 13):
    Expected : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:()\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:()\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:()\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}\x08\x08\ruser> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
FAILURES: 2
make: *** [test^perl^step0] Error 2

from mal.

Okay, let's look a little closer at what is happening for the output from the first line and split at newlines/carriage returns:

;:()\x08\x08\r
user> ;:() []\x08\x08\r
user> ;:() []{}\x08\x08\r
user> ;:() []{}"\'*\r\n
;:() []{}"\'*

And now the longer one (with the regular alphanumeric section shortened to just ZZZ):

ZZZ (;:()\x08\x08\r
user> ZZZ (;:() []\x08\x08\r
user> ZZZ (;:() []{}\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:()\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:() []\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:() []{}\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:() []{}"\'* ;:()\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:() []{}"\'* ;:() []\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:() []{}"\'* ;:() []{}\x08\x08\r
user> ZZZ (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\n
ZZZ (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'

So here is what it looks like is happening to me: when your readline is reading from stdin and it comes to a certain character sequences (in this case "()", "[]", and "{}", it is backspacing twice (\x08), and then carriage returning (\r by itself) back to the beginning of the same line and starting again. Each time it does that it gets a little farther (until the next character sequence like that). The output from stdout appears to be unaffected (the final line). Again, I have no idea why it would be doing that, but that's what it appears to be doing to me.

from mal.

@vi I had an idea, can you use the script command in the same terminal and then paste this string into it?

script recording.txt
#hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{}"'*)
<Ctrl-D>

That should give you file called recording.txt that has exactly what was output to your terminal (including control codes an everything). Then you can do hexdump -C recording.txt to see/post the literal output here. It would be interesting to compare to your normal terminal behavior to see if it has the same behavior.

from mal.

v@l:00:36:05:~/src/git/mal$ script recording.txt
Script started, file is recording.txt
v@l:00:36:07:~/src/git/mal$ #hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{}"'*)
v@l:00:36:17:~/src/git/mal$ exit
Script done, file is recording.txt
v@l:00:37:08:~/src/git/mal$ base64  recording.txt 
U2NyaXB0IHN0YXJ0ZWQgb24gVGh1IDE5IE1hciAyMDE1IDEyOjM2OjA3IEFNIEZFVAobXTA7bWFs
IH4vc3JjL2dpdCAodmkpBxtbRxtbMDE7MzJtdkBsOjAwOjM2OjA3On4vc3JjL2dpdC9tYWwkIBtb
MDE7MG0jaGVsbG8gd29ybGQgYWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXogQUJDREVGR0hJSktM
TU5PUFFSU1RVVldYWVogMDEyMzQ1Njc4OSAoOzooKQgIG1tDG1tDIFtdCAgbW0MbW0N7fQgIG1tD
G1tDIicqIDs6KCkICBtbQxtbQyBbXQgIG1tDG1tDe30ICBtbQxtbQyInKiA7OigpCAgbW0MbW0Mg
W10ICBtbQxtbQ3t9CAgbW0MbW0MiJyopDQobXTA7bWFsIH4vc3JjL2dpdCAodmkpBxtbRxtbMDE7
MzJtdkBsOjAwOjM2OjE3On4vc3JjL2dpdC9tYWwkIBtbMDE7MG1leGl0DQoKU2NyaXB0IGRvbmUg
b24gVGh1IDE5IE1hciAyMDE1IDEyOjM2OjE5IEFNIEZFVAo=

from mal.

Maybe it's because of set blink-matching-paren on in /etc/inputrc?

If it is and there's no wat to make it ignore config, then it shows that readline is an essentially user-oriented library which should be assumed customized and tested only separately (not in bundle with other tests).

from mal.

Aha! Well, that's certainly part of it. I added that to ~/.inputrc and was able to replicate that same behavior running the rust step0 tests. I've pushed a change to make runtest.py ignore inputrc (setting INPUTRC to /dev/null). And doing that made it work for me even with the setting in ~/.inputrc.

Let me know how that goes. As a side benefit, the tests should at least fail faster because the way that blink-matching-paren work is using output delays and simulated cursor movement to "blink".

from mal.

Now testing Perl and Rust works.
(Perl fails in stepA for unrelated problems).

from mal.

@vi yep, stepA contains two optional sections. One is optional but needed for self-hosting. The other is just extras. There are a few implementations that have some stepA failures in that section. I'm going to go ahead and resolve this since the main failures seem to have been addressed.

from mal.

@kanaka The problem seems appear again. Please try "make test^guile^step0" in my pull request.

from mal.

@NalaGinrut this is because the guile wrapper around GNU readline adds a custom paren bouncing feature. See "init_bouncing_parens" in guile-readline/readline.c in the guile sources.

Here is a diff that passes all tests and works with readline enabled:

diff --git a/guile/readline.scm b/guile/readline.scm
index 7845887..95501b0 100644
--- a/guile/readline.scm
+++ b/guile/readline.scm
@@ -24,6 +24,7 @@

 (setenv "GUILE_HISTORY" mal-history)

+(readline-set! bounce-parens 0)
 (activate-readline)

 ;;(define (readline prompt)

However, I do notice that readline history isn't hooked up right. It's adding the result of the expression to the history (rather that the form that was entered), it's not storing it in the history file, and it only works for stepA. I suggest that you provide a wrapper function in readline.scm that adds to history and stores to the history file and then call that from the steps.

from mal.