export TEST='bonjour je suis'
echo yo > $TEST

is an error

Add test for case insensitivity for all builtins. Already tested echo and regular commands, need every other builtins.

As mentioned in #12

The user should be able to put custom test suites in a minishell_test directory in his project's directory.

After pip install minishell-test the user should be able to run minishell_test as a command or python -m minishell_test.

The default path to minishell would be . with an option to change it.

The user could add his tests in a test directory at the root of his project.

Although having the user pip instead of installing with git clone would make the tests more oppace (can't see the source directly). People who would be intreasted in modifying the test will have to clone the source and install them with pip install -e.

Pass the command through SDTIN and ignoring any output that is defined by the user as his prompt.

Could have prompt format {abs_path}$ would be a prompt starting with the absolute path prefixed by $ or {dir_basename} > the basedir of the path prefixed by > . Other format could support username, hostname, etc..

Configured in the configuration file, with an arguement or via a environmnent variable (like MINISHELL_TEST_PROMPT).

Fields a separated by spaces and tabs, should add test for \f \v \r \n and other weird ones

Hey! 👋 @cacharle

Great project to test if a shell follows specifications. 👍
For fun, I decided to test the real executable bash with this tool but somehow there are test failures, is bash not supposed to pass all the tests at least (as probably bash would do a lot more than what a Minishell project is supposed to do)? To be precise there are 621 PASS and 364 FAIL.

$ bash --version
GNU bash, version 5.1.8(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Steps to reproduce

mkdir Minishell ; cd Minishell
cp /bin/bash minishell
touch minishell_test.cfg

The content of minishell_test.cfg is the default from the documentation: https://minishell-test.readthedocs.io/en/latest/config.html, with exception that make = false.

$ cat minishell_test.cfg
[minishell_test]
bonus = false
exec_name = minishell
make = false
make_args = MINISHELL_TEST_FLAGS=-DMINISHELL_TEST
pager = false
pager_prog = less
log_path = minishell_test.log
check_error_messages = true
end_command_with_linefeed = true

[shell]
available_commands =
    rmdir
    env
    cat
    touch
    ls
    grep
    sh
    head
path_variable = {shell_available_commands_dir}

[shell:reference]
path = /bin/bash
args =

[timeout]
test = 0.5
leaks = 10

Then execute minishell_test.

Some explanation of how this project work, and what specification it tests actually would be helpful, thanks! 😄

EDIT: Most of the tests failed because of the wrong path, using exec name = /fullpath/minshell instead of only minishell result in 926 PASS and 59 FAIL, so this is a lot better, but there are still 59 tests failures and I guess that shouldn't be the case.
First failing test from minishell_test.log:

|> WITH >\n
|---------------------------------------EXPECTED--------------------------------
/home/divlo/Downloads/Minishell/minishell: syntax error near unexpected token `newline'$
|---------------------------------------ACTUAL----------------------------------
/home/divlo/Downloads/Minishell/minishell: -c: syntax error near unexpected token `newline'$
/home/divlo/Downloads/Minishell/minishell: -c: `>'$
================================================================================

Some important information for test are truncated.

e.g SHLVL=' 0000000000000000000000000000000001 the 1 is truncated.

Some truncation on long meaningless text are useful.

Some implementation of the -c option use termcaps and expect the command passed to it to end with a linefeed.

It's bash: line 1: ... or bash: line 0: (depending on the version)

Clear documentation site instead of cluttering the README with a ton of information.

Somewhere that isn't the current directory

We can select the suite to run but it would be nice to exclude some suite.

I have a commented test suite for glob (e.g *.c), it should be disabled by default in the config.
Same thing if someone doesn't care about the SHLVL tests, they should be able to disable them without having to comment out the test suite.

Those 3 tests segfaults in the tester but they doesn't do that when i try ./minishell -c ">" for an example in my program.

The user should be able to change the default configuration in a minishell_test/config.py in his project directory.