nesvoboda / socrates Goto Github PK

View Code? Open in Web Editor NEW

64.0 64.0 3.0 107 KB

A small testing framework for 42's philosophers

Python 99.16% Makefile 0.84%

socrates's Introduction

socrates

A small testing framework for 42's philosophers.

New: now includes Delay-o-meter, a tool to measure sleep inaccuracy

Acknowledgements

Big thanks to:

solaldunckel for helping to test this project!

cacharle for their awesome pull requests that are pushing this project forward!

What are the tests?

Socrates tests two things.

In PERFORMANCE tests, it times how long your program runs in a test case where no philosophers should die (like 550 210 210). By default, your program should hold on for 40 seconds.

In a DEATH TIMING test, we make a philosopher die (sorry!) (example: 100 500 500, a philosopher will die at 100ms). The output of your program is then parsed to measure how soon your program reported that.

If your delay is more than 10ms, the test is failed!

Delay-o-meter

Different machines perform sleeps with different accuracy. Socrates reports the average delay the machine will add to a 200-millisecond sleep. This can help make sure other stuff running on the computer doesn't interfere with Philosopher timings.

You can also use a standalone tool and check different machines:

python3 delay_o_meter.py

How to interpret the result?

The common-sense standard of a good Philosophers is less than 10ms of delay per one eat-sleep-think cycle (example: 310 150 150 should run forever).

My personal opinion is that, therefore, a machine must lose less than 3ms on average for tests to be accurate. Let me know what you think about that!

CPU load detector

Another way of making sure nothing prevents your philosophers from working properly, Socrates checks the load of your CPU. If it's overloaded, it will show a message.

Here's an article that explains why this is important, especially on a VM.

Installation

Requirements: python 3.6+, psutil

Clone the repo

git clone https://github.com/nesvoboda/socrates
cd socrates

Install the prerequisites

pip3 install -r requirements.txt

If you're in a 42 campus, run this instead:

python3 -m pip install -r requirements.txt --user

If installation fails on Linux, try this:

sudo apt-get install python3.7-dev

Thanks to Mazoise for this suggestion!

Configuration

You can edit the variables N_LONG_TESTS and LONG_TEST_LENGTH to determine how long the preformance tests will last. My standard is 3 consecutive runs of at least 40 seconds.

Running the tests

python3 socrates.py <path to project folder>

socrates's People

Contributors

Stargazers

Watchers

Forkers

cacharle goat095 jgiron42

socrates's Issues

Systemic crash on DEATH TIMING

DEATH TIMING

0ms: philo 1 is thinking
0ms: philo 1 has taken a fork
0ms: philo 1 has taken a fork
0ms: philo 1 is eating
0ms: philo 3 is thinking
0ms: philo 3 has taken a fork
0ms: philo 2 is thinking
200ms: philo 1 is sleeping
200ms: philo 2 has taken a fork
200ms: philo 3 has taken a fork
200ms: philo 3 is eating
300ms: philo 1 is thinking
311ms: philo 1 died
Traceback (most recent call last):
  File "/home/maplepy/git/cc/philosophers/philo/socrates/socrates.py", line 302, in <module>
    exit(socrates(args.path, args.philo))
  File "/home/maplepy/git/cc/philosophers/philo/socrates/socrates.py", line 258, in socrates
    test_program(f"{bin_path}/philo/philo")
  File "/home/maplepy/git/cc/philosophers/philo/socrates/socrates.py", line 177, in test_program
    if run_starvation_measures(binary) is False:
  File "/home/maplepy/git/cc/philosophers/philo/socrates/socrates.py", line 151, in run_starvation_measures
    results.append(measure_starvation_timing(binary))
  File "/home/maplepy/git/cc/philosophers/philo/socrates/socrates.py", line 121, in measure_starvation_timing
    start_time = int(first_line[:separator_index])
ValueError: invalid literal for int() with base 10: '0ms:'

Requirements

Hi, I had a small issue installing requirements.
To install psutil you need a few more packages, at least for python 3.7 (I don't know for other versions)

sudo apt-get install python3.7-dev

solved the problem for me.

Thanks for your tester, it's very useful !

Add argument to test a specific philosophers

./socrates.py -p1 to only test philo_one and so on.

Add progress indicator during performance tests

Having to wait 40 seconds without any feedback is frustrating. A timer with the current number of seconds or a progress bar would make the experience better.

Subprocess is not cleaned up

Hello! Cool project, thank you.

After running the tests

python3.7 socrates.py -p 1 /philosophers

Grep'ing processes with the name philo

ps aux | head -n +1
ps aux | grep "philo"

Outputs:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user42     7265  0.4  0.0  47680   856 pts/1    Sl   20:32   0:00 /philosophers/philo_one/philo_one 4 311 150 150
user42     7273  0.4  0.0  47680   876 pts/1    Sl   20:32   0:00 /philosophers/philo_one/philo_one 4 311 150 150
user42     7280  0.4  0.0  47680   888 pts/1    Sl   20:32   0:00 /philosophers/philo_one/philo_one 4 311 150 150
user42     7288  0.4  0.0  55876   948 pts/1    Sl   20:33   0:00 /philosophers/philo_one/philo_one 5 600 150 150
user42     7297  0.4  0.0  55876   876 pts/1    Sl   20:33   0:00 /philosophers/philo_one/philo_one 5 600 150 150
user42     7305  0.4  0.0  55876   836 pts/1    Sl   20:33   0:00 /philosophers/philo_one/philo_one 5 600 150 150
user42     7386  0.0  0.0  16952  1060 pts/1    S+   20:33   0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox philo_one

I think this is happening because the program is running in a shell and process.kill() seems to kill only the shell.

A posible fix is to import shlex and in assert_runs_for_at_least() set shell to False

process = subprocess.Popen(shlex.split(command), stdout=subprocess.DEVNULL, shell=False)

This will work for philo_one and philo_two because they use threads.

philo_three it's creating multiple subprocesses, so they need to be cleaned up before returning.

Tested on Xubuntu 18.04 with python3.7

Edit:

Just saw the function processes_still_running() which should clean the processes and I don't know why it's not working on my machine, will test it more since I'm currently working on philosophers project.

when testing the current philo_bonus, throws error

It throws this exception because it can't get the output of the command.

Traceback (most recent call last):
  File "socrates/socrates.py", line 302, in <module>
    exit(socrates(args.path, args.philo))
  File "socrates/socrates.py", line 263, in socrates
    test_program(f"{bin_path}/philo_bonus/philo_bonus")
  File "socrates/socrates.py", line 177, in test_program
    if run_starvation_measures(binary) is False:
  File "socrates/socrates.py", line 151, in run_starvation_measures
    results.append(measure_starvation_timing(binary))
  File "socrates/socrates.py", line 114, in measure_starvation_timing
    if data[-1] == "\0":
IndexError: string index out of range

The reason is that the new bonus of the philosophers project says every philosopher has to be a process. So when it tries to get the output of ./philo_bonus, it doesn't capture the output from it's child processes.

So the string data becomes an empty string and throws an exception.

I couldn't really think of a way of capturing output from child processes but I will look into that.

Add option to select/change infinite test

Change the tested arguments from the command line (changing in the file is too cumbersome).

e.g: ./socrates ../philosophers 4 410 200 200

Select the infinite tests, if I know that one passes and not the other, I only want to test one of them. It's a waste of time.

Error in installation command (42) and error in death timing

Hello guys ! great work but two issues:

for the installation command line in 42:
python3 -m pip install psutil --user and not python3 -m pip install psutil -user

And death timing doesn't work for me in the school.
However my printing is good:
00000000001 3 is eating
the logs:

Traceback (most recent call last):
  File "socrates.py", line 253, in <module>
    exit(socrates(bin_path))
  File "socrates.py", line 235, in socrates
    test_program(f"{bin_path}/philo_one/philo_one")
  File "socrates.py", line 177, in test_program
    if run_starvation_measures(binary) is False:
  File "socrates.py", line 151, in run_starvation_measures
    measure_starvation_timing(binary, results)
  File "socrates.py", line 125, in measure_starvation_timing
    separator_index = pattern.search(last_line).start()
AttributeError: 'NoneType' object has no attribute 'start'

Potential death time parsing bug

I had this error once with philo_one:

DEATH TIMING

[2 MS] [3 MS] [2 MS] [2 MS] Traceback (most recent call last):
  File "./socrates.py", line 281, in <module>
    exit(socrates(args.path, args.philo))
  File "./socrates.py", line 242, in socrates
    test_program(f"{bin_path}/philo_one/philo_one")
  File "./socrates.py", line 157, in test_program
    if run_starvation_measures(binary) is False:
  File "./socrates.py", line 131, in run_starvation_measures
    measure_starvation_timing(binary, results)
  File "./socrates.py", line 108, in measure_starvation_timing
    death_time = int(last_line[:separator_index].strip("\0"))
ValueError: invalid literal for int() with base 10: 'dyld:'

When I run the test on my own I have the following output:

❯ ./philo_one 3 310 200 100 | cat -e
0 2 has taken fork$
0 2 has taken fork$
0 2 is eating$
2 3 has taken fork$
200 2 is sleeping$
200 3 has taken fork$
200 3 is eating$
200 1 has taken fork$
300 2 is thinking$
311 2 died$

I never had this error before.

Don't stop the test on fail

When a fail occurs the test should continue the next tests to show more feed back to the user.

For the infinity test, the user would be able to see if the fail is an isolated incident or if it fails every time

ValueError: invalid literal for int() with base 10

Hi,
First, thanks for your work it's great help.
My only concern is when socrates.py tests death timing.
Do you know how to correct this line in socrates.py ?
death_time = int(last_line[: last_line.find(" ")])

See here the last line :

 DEATH TIMING

Running ../..//philo_one/philo_one 3 310 200 100
ran ../..//philo_one/philo_one 3 310 200 100
Traceback (most recent call last):
  File "socrates.py", line 217, in <module>
    exit(socrates(bin_path))
  File "socrates.py", line 198, in socrates
    test_program(f"{bin_path}/philo_one/philo_one")
  File "socrates.py", line 140, in test_program
    if run_starvation_measures(binary) is False:
  File "socrates.py", line 119, in run_starvation_measures
    measure_starvation_timing(binary, results)
  File "socrates.py", line 95, in measure_starvation_timing
    death_time = int(last_line[: last_line.find(" ")])
ValueError: invalid literal for int() with base 10: '311\t1'

Thx