rejeep / ert-runner.el Goto Github PK

On my emacs (GNU Emacs 24.5.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 2015-07-09 on herakleitos) the dot reporter doesn't print anything before all the tests are over. Shouldn't it print a dot as soon as the test passes?

I have been hacking on a junit reporter, and made the following discoveries:

1. it's not possible to set ert-runner-reporters-path from the command line, nor can you load any file that sets that variable before tests are run. It would be great to simply have this be settable via the command line.

2. ert-runner logs various messages to stdout (such as backtraces) that clash with output from the custom reporter. I think all output (aside from debug or maybe verbose level) should be driven from inside the reporters, to allow them full control.

The hacks I needed to employ to get around this without forking and patching were pretty brutal :) So much so that it's pretty clear to me it would be beneficial to fix this upstream. I can probably cobble together a PR but I wanted to solicit feedback first. Perhaps the JUnit reporter could also simply be included – it's a fairly trivial hook.

For entertainment, here's what I had to do to work around this:

• create a script that I can run with cask emacs
• (setq commander-ignore t) to stop ert-runner from seizing control
• (load "ert-runner") squashing error messages on load
• set ert-runner-reporters-path and ert-runner-reporter-name
• redefine ert-runner-message to be a no-op
• invoke (ert-runner/run)

If I want to use a different version of a package in my tests (in my case I want to use a custom org-mode version) I have to perform some processing of the load-path in my test helper I found out after some debugging.

https://github.com/remyhonig/elfeed-org/blob/master/test/test-helper.el

;; Remove standard org-mode paths so I can supply them with my own
;; using the "cask exec ert-runner -L org-mode/lisp" command
(setq load-path (-remove (lambda (path) (s-ends-with? "lisp/org" path)) load-path))
(message "Org version: %s on Emacs version: %s" (org-version) (emacs-version))

The -L option of ert-runner appends the path and I expected it to work the same way as the -L option of emacs.

01c9442

But while reading the CHANGELOG for emacs I found out the -L option should prepend to the load-path

http://www.gnu.org/software/emacs/news/NEWS.24.5

** The -L option, which normally prepends its argument to load-path,
will instead append, if the argument begins with `:' (or `;' on MS Windows;
i.e., `path-separator').

My request is to change the behaviour of the -L option to match the one of emacs (so to prepend instead append) . If you don't have the time I'd be happy to provide a PR. Just let me know if you do.

Plains loading of the library should depend on setting of ERT_RUNNER_ARGS

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
string-match(" " nil 0)
split-string(nil " " nil)
s-split(" " nil)
(-reject (quote s-blank?) (s-split " " (getenv "ERT_RUNNER_ARGS")))

Add support for project specific configuration file (.ert-runner). For example:

-l test/ert-loader.el

$ cask exec ert-runner run -p parse-cppcheck 
Option `-p` requires argument

But:

$ cask exec ert-runner run -p cppcheck
Loading erlang-skels...
Running 6 tests (2013-08-09 17:20:42+0200)
   failed  1/6  checker-c/c++-cppcheck-error
   failed  2/6  checker-c/c++-cppcheck-multiple-checks
   failed  3/6  checker-c/c++-cppcheck-style
   failed  4/6  checker-c/c++-cppcheck-style-suppressed
   failed  5/6  checker-c/c++-cppcheck-warning
   passed  6/6  flycheck-parse-cppcheck

Ran 6 tests, 6 results as expected (2013-08-09 17:20:42+0200)
5 expected failures

Probably a bug in commander?

I agree with ert-runner's usage of convention over configuration to locate test files, but I think requiring all projects to move their tests to a test directory is fairly strict. Since it's extremely likely that a file named -test.el or -tests.el has ERT test definitions, I think it would be reasonable and convenient to extend ert-runner's concept of a test file to any file matching the regex -tests?.el$ (while still matching all files in a test directory, which would now be optional). This already popular convention should help make ert-runner easier to migrate to for most projects.

Most Emacs projects I come across only have one implementation file, so the single test file convention is fairly useful to avoid the creation of extra directories. It's also worth noting that some popular test frameworks for other languages (like Jest for JavaScript) already use a similar convention for naming test files, which could make onboarding easier. Jest specifically uses a convention of .test.jsx?$ files or files in a __tests__ directory, but this concept also translates nicely to .tests?.el$ and test for Emacs LISP files. I would expect an implementation of this naming convention in ert-runner to be fully back compatible, accepting all files in test without doing further filename filtering on them.

Today I have opened issues against 69 packages that come with a file test-helper.el that provides the feature test-helper. This was premature and I apologize to everyone affected.

I have now been made aware that ert-runner loads that file if it exists, so the file should not be renamed to <package>-test-helper.el as I suggested in those issues. However I still think these elisp files should not provide the feature test-helper because they are not actual libraries that are loaded using require but elisp files that contain testing helpers.

ert-runner does not use require to load the test-helper.el of the package being tested. It uses ert-runner--load, which is a wrapper around load, which loads a file using the path to it, unlike require which locates the file matching a feature on the load-path. See ert-runner.el#L198-L201 and ert-runner.el#L165-L168.

@rejeep, as the author of ert-runner, do you agree that the test-helper.el files should not provide a feature?

others, until we have come to a conclusion here, you should not rename test-helper.el and also delay dropping the (provide 'test-helper).

This is a feature request.

Just like emacs, I think it's useful to add an -L option to ert-runner, thus we can specify load-path instead of load files manually by ert-runner -l file1 file2.

How do you think about it?

Ert runner have some runtime dependencies that will pollute the test environment. Run tests in subprocess to avoid this.

Since the latest MELPA build 20130904.2140 ert-runner loads the test/.dir-locals.el file, causing a build error.

The error only occurs when running ert-runner without any arguments. For this reason, my attempt to obtain a stracktrack with --debug failed.

However, inserting (setq debug-on-error t) at the top of ert-runner.el eventually gave me the following stacktrace:

Debugger entered--Lisp error: (invalid-function (emacs-lisp-mode (eval push default-directory flycheck-emacs-lisp-load-path)))
  ((emacs-lisp-mode (eval push default-directory flycheck-emacs-lisp-load-path)))
  eval-buffer(#<buffer  *load*-563283> nil "/Users/swiesner/Developer/Projects/flycheck/test/.dir-locals.el" nil t)  ; Reading at buffer position 183
  load-with-code-conversion("/Users/swiesner/Developer/Projects/flycheck/test/.dir-locals.el" "/Users/swiesner/Developer/Projects/flycheck/test/.dir-locals.el" nil t)
  load("/Users/swiesner/Developer/Projects/flycheck/test/.dir-locals.el" nil :nomessage)
  ert-runner--load("/Users/swiesner/Developer/Projects/flycheck/test/.dir-locals.el")
  -each(("/Users/swiesner/Developer/Projects/flycheck/test/.dir-locals.el" "/Users/swiesner/Developer/Projects/flycheck/test/automatic-checking-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/builtin-checkers-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/checker-api-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/config-file-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/customization-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/deferred-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/definition-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/documentation-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/error-api-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/error-display-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/error-list-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/error-messages-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/error-parsers-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/global-mode-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/mode-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/navigation-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/option-filters-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/overlay-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/process-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/selection-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/test-helper.el" "/Users/swiesner/Developer/Projects/flycheck/test/utility-test.el" "/Users/swiesner/Developer/Projects/flycheck/test/version-test.el") ert-runner--load)
  (let* ((el-tests-fn (function (lambda (file) (if tests (---truthy\? (let ... ... needle)) (s-matches\? "-test.el$" file))))) (test-files (f-files (f-expand ert-runner-test-path) el-tests-fn)) (test-helper (f-expand "test-helper.el" ert-runner-test-path))) (-each ert-runner-load-files (function ert-runner--load)) (if (f-exists\? test-helper) (ert-runner--load test-helper)) (-each test-files (function ert-runner--load)) (ert-run-tests-batch-and-exit ert-runner-selector))
  ert-runner/run("")
  apply(ert-runner/run "")
  commander--handle-command((""))
  commander-parse((""))
  (if commander-parsing-done nil (commander-parse (or commander-args (cdr command-line-args-left))))
  (progn (setq commander-default-config nil) (setq commander-options nil) (setq commander-commands nil) (setq commander-name nil) (setq commander-description nil) (setq commander-default-command nil) (setq commander-no-command nil) (setq commander-parsing-done nil) (-each (quote ((name "ert-runner") (description "Opinionated Ert testing workflow") (config ".ert-runner") (default ert-runner/run) (option "--help, -h" "Show usage information" ert-runner/usage) (option "--pattern <pattern>, -p <pattern>" "Run tests matching pattern" ert-runner/pattern) (option "--load <*>, -l <*>" "Load files" ert-runner/load) (option "--debug" "Enable debug" ert-runner/debug) (option "--script" "Run Emacs as a script/batch job (default)" ignore) (option "--no-win" "Run Emacs without GUI window" ignore) (option "--win" "Run Emacs with full GUI window" ignore) (command "init [name]" "Create new test project (optional project name)" ert-runner/init) (command "help" "Show usage information" ert-runner/usage))) (function (lambda (form) (cond ((eql (car form) (quote option)) (progn (let* ... ...))) ((eql (car form) (quote command)) (progn (let* ... ...))) ((eql (car form) (quote parse)) (progn (let* ... ... ...))) ((eql (car form) (quote name)) (progn (let* ... ...))) ((eql (car form) (quote description)) (progn (let* ... ...))) ((eql (car form) (quote config)) (progn (let* ... ...))) ((eql (car form) (quote default)) (progn (let* ... ...))) (t (error "Unknown directive: %S" form)))))) (if commander-parsing-done nil (commander-parse (or commander-args (cdr command-line-args-left)))))
  eval-buffer(#<buffer  *load*> nil "/Users/swiesner/Developer/Projects/flycheck/.cask/24.3.50.1/elpa/ert-runner-20130904.2140/ert-runner.el" nil t)  ; Reading at buffer position 4703
  load-with-code-conversion("/Users/swiesner/Developer/Projects/flycheck/.cask/24.3.50.1/elpa/ert-runner-20130904.2140/ert-runner.el" "/Users/swiesner/Developer/Projects/flycheck/.cask/24.3.50.1/elpa/ert-runner-20130904.2140/ert-runner.el" nil t)
  load("/Users/swiesner/Developer/Projects/flycheck/.cask/24.3.50.1/elpa/ert-runner-20130904.2140/ert-runner.el" nil t t)
  command-line-1(("-scriptload" "/Users/swiesner/Developer/Projects/flycheck/.cask/24.3.50.1/elpa/ert-runner-20130904.2140/ert-runner.el"))
  command-line()
  normal-top-level()

As you can see, somehow .dir-locals.el ends up in the test-files list in ert-runner/run, but again, only if no command line argument is passed.

I am not sure that ert-runner is really at fault here. It is probably a command bug. It seems that an empty string somehow ends up in the tests if no argument is given.

See https://github.com/rejeep/shut-up.el

The "nonexistent files" test fails for me because Emacs now uses Unicode quotes by default:

Then I should see error "/missing-test.el` does not exist"
      Test failed: ((should (s-contains\? expected ert-runner-error)) :form (s-contains\? "/missing-test.el` does not exist" "‘/path/to/ert-runner.el/features/project/test/missing-test.el‘ does not exist.
      ") :value nil)

Either you need to bind text-quoting-style or check for either ASCII or Unicode quotes.
(The quote itself is also incorrect; it's an opening quote, not a closing quote.)

See #4 (comment) for discussion.

Both locally and on Travis CI:

17 scenarios (9 failed, 8 passed)
70 steps (9 failed, 4 skipped, 57 passed)
make[1]: *** [ecukes] Error 1
make: *** [test] Error 2

See failed build on Travis for details.

ert-runner uses the internal function ert--print-backtrace. Previously, that function had the signature (backtrace), now (Emacs 26) the signature is (backtrace do-xrefs). This means that printing backtraces fails with Emacs 26. ert-runner could work around this by either catching wrong-number-of-arguments or checking the signature using help-function-arglist.

Hello.

I've tried launching ert-runner on Windows and it does not work. I get:

cask exec: error: [Errno 2] No such file or directory

Adding --debug does not add any new output.

There's a ert-runner bash script, in my .cask project folder, but running it manually using msys bash, does not work.

I've been using ert-runner for the longest time and I always include this piece of code into the test-helper.el file

(let ((dir (f-parent (f-dirname (f-this-file)))))
  (add-to-list 'load-path dir))
(require '<MY-PACKAGE>)

What this does essentially is it adds ../ on the load-path. I can never get things to work without it as neither cask load-path would report the package's path there nor require finds the files.

However, I've noticed projects which don't do this and I can't figure out how/where to configure things to simply include the directory with Cask on the load-path (in fact, all the projects in the example section work just fine like that).

What do I need to do? :O

It appears that the function signature of the built-in ert--print-backtrace changed somewhere between Emacs 25.3 and Emacs 26.1, adding a second required argument. Consequently, some call within the plumbing of ERT is failing when using ert-runner in Emacs 26.1.

Note that if all of your tests are passing, you will encounter no problems. However, if you have a failing test, instead of seeing some helpful test failure information, you instead see (wrong-number-of-arguments (2 . 2) 1). With the --debug flag, you will see the native Emacs backtrace ending on the frame ert--print-backtrace(((....

Here is what it looks like in Emacs 26.1:

% EMACS=/usr/local/Cellar/emacs/26.1_1/Emacs.app/Contents/MacOS/EmacS cask exec ert-runner --debug
.Test commit-in-unsaved-buffer-fails backtrace:


Debugger entered--Lisp error: (wrong-number-of-arguments (2 . 2) 1)
  ert--print-backtrace(((t signal (error ("Periodic Commit cannot commit a file with no filename!")) nil) (t error ("Periodic Commit cannot c

<hundreds of lines of backtrace snipped>

And here is what is expected, which is running the same test suite with one failing test in Emacs 25.3. Note that most of the debug information here is relevant to the test suite I'm running, which is not important for this particular bug (and this is even rather useless since it is the expected output from the program):

% EMACS=/usr/local/Cellar/emacs/25.3/Emacs.app/Contents/MacOS/EmacS cask exec ert-runner
.Test commit-in-unsaved-buffer-fails backtrace:

  (if (not (buffer-file-name)) (error "Periodic Commit cannot commit a
  pcmm--commit(t)
  pcmm-commit()
  apply(pcmm-commit nil)
  (setq value-16 (apply fn-14 args-15))
  (unwind-protect (setq value-16 (apply fn-14 args-15)) (setq form-des
  (if (unwind-protect (setq value-16 (apply fn-14 args-15)) (setq form
  (let (form-description-18) (if (unwind-protect (setq value-16 (apply
  (let ((value-16 (quote ert-form-evaluation-aborted-17))) (let (form-
  (let ((fn-14 (function pcmm-commit)) (args-15 (list))) (let ((value-
  (save-current-buffer (set-buffer (get-buffer-create "empty buffer"))
  (pcmmt-create-proj "project" (save-current-buffer (set-buffer (get-b
  (let* ((pcmmt-dir (pcmmt-create-proj "project" (save-current-buffer
  (unwind-protect (let* ((pcmmt-dir (pcmmt-create-proj "project" (save
  (lambda nil (unwind-protect (let* ((pcmmt-dir (pcmmt-create-proj "pr
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  ert-run-test([cl-struct-ert-test commit-in-unsaved-buffer-fails "You
  ert-run-or-rerun-test([cl-struct-ert--stats (and t) [[cl-struct-ert-
  ert-run-tests((and t) (lambda (event-type &rest event-args) (cond ((
  ert-runner/run-tests-batch((and t))
  (let ((stats (ert-runner/run-tests-batch selector))) (kill-emacs (if
  ert-runner/run-tests-batch-and-exit((and t))
  (if ert-runner-verbose (ert-runner/run-tests-batch-and-exit ert-runn
  (let ((test-files (ert-runner--test-files tests)) (test-helper (f-ex
  ert-runner/run()
  apply(ert-runner/run nil)
  commander--handle-command(nil)
  commander-parse(nil)
  (if commander-parsing-done nil (commander-parse (or commander-args (
  eval-buffer(#<buffer  *load*> nil "/Users/airborne/dotfiles/configs/
  load-with-code-conversion("/Users/airborne/dotfiles/configs/emacs.d/
  load("/Users/airborne/dotfiles/configs/emacs.d/periodic-commit-minor
  command-line-1(("-scriptload" "/Users/airborne/dotfiles/configs/emac
  command-line()
  normal-top-level()

Test commit-in-unsaved-buffer-fails condition:

    (error "Periodic Commit cannot commit a file with no filename!")

<snip>

One way to locally fix this is to force ert-compat.el to load, since it brings along its own ert--print-backtrace that expects a single argument, which overrides the built-in one.

As a new user of both ERT and ert-runner, I banged my head against my desk for hours trying to figure out why this one test was getting stuck on this backtrace call until I discovered that it is because the test was failing for a valid reason, but spitting out this useless internal backtrace instead.

Let me know if I can be helpful at all; I just happen to have these two versions of Emacs installed, so I haven't tracked down the precise change that brought this about, nor have I picked through the guts of ert-runner to try to find the source of the problem.

When the test passes, I don't really care about any messages the test produces. I would just want to see all the dots. If the test fails and we want to use some quick&dirty printf debugging, the tester should capture the output and print it.

Could we somehow wrap the test with shut-up or similar (ideally capturing the output to a string so we can later dump it in case of failure). I'm not familiar with the internal test-running logic.

The Ert reporter is broken in Emacs 24.4. This is only an issue with byte compiled files. Using it will give the output: Invalid time specification

hi
i've got a unit test 1 which fails like that :

cask exec ert-runner -t current
[dionysos] Cleanup path
[dionysos] Load library from /home/nlamirault/Perso/dionysos/dionysos.el
[dionysos-process] Create dionysos mpg123 (/home/nlamirault/Perso/dionysos/test/resources/Roulement_tambour-01.mp3) (lambda nil (message ok))
[dionysos-process] Status : nil
Test test-dionysos-process-can-start-process backtrace:

  (let ((file (f-join dionysos-testsuite-dir "resources/Roulement_tamb
  (let ((default-directory dionysos-source-dir)) (cleanup-load-path) (
  (condition-case nil (let ((default-directory dionysos-source-dir)) (
  (unwind-protect (condition-case nil (let ((default-directory dionyso
  (lambda nil (unwind-protect (condition-case nil (let ((default-direc
  #[0 "\306\307!r\211q\210\310\311\312\313\314\315!\316\"\317\320%DC
  funcall(#[0 "\306\307!r\211q\210\310\311\312\313\314\315!\316\"\31
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  #[0 "r\304 q\210\305 )\306\307\310\311\312\313!\314\"\315\316%DC\2
  funcall(#[0 "r\304 q\210\305 )\306\307\310\311\312\313!\314\"\315\
  ert-run-test([cl-struct-ert-test test-dionysos-process-can-start-pro
  ert-run-or-rerun-test([cl-struct-ert--stats (and t (or (tag current)
  ert-run-tests((and t (or (tag current))) (lambda (event-type &rest e
  ert-runner/run-tests-batch((and t (or (tag current))))
  (let ((stats (ert-runner/run-tests-batch selector))) (kill-emacs (if
  ert-runner/run-tests-batch-and-exit((and t (or (tag current))))
  (if ert-runner-verbose (ert-runner/run-tests-batch-and-exit ert-runn
  (let ((test-files (ert-runner--test-files tests)) (test-helper (f-ex
  ert-runner/run()
  apply(ert-runner/run nil)
  commander--handle-command(nil)
  commander-parse(("-t" "current"))
  (if commander-parsing-done nil (commander-parse (or commander-args (
  (progn (setq commander-default-config nil) (setq commander-options n
  eval-buffer(#<buffer  *load*> nil "/home/nlamirault/Perso/dionysos/.
  load-with-code-conversion("/home/nlamirault/Perso/dionysos/.cask/24.
  load("/home/nlamirault/Perso/dionysos/.cask/24.5/elpa/ert-runner-201
  command-line-1(("-scriptload" "/home/nlamirault/Perso/dionysos/.cask
  command-line()
  normal-top-level()

Test test-dionysos-process-can-start-process condition:

    (invalid-function
     (hook hook))

F

Ran 1 test in 0.045 seconds
1 unexpected results:
   FAILED  test-dionysos-process-can-start-process

but function dionysos--create-process seems works correctly if i use it into an IELM buffer.

ELISP> (dionysos--create-process "mpg123" "mpg123" (list "/home/nlamirault/Perso/dionysos/test/resources/Roulement_tambour-01.mp3") (lambda () (message "OK")))
(lambda
  (&rest --cl-rest--)
  (apply
   '(lambda
      (G66 process event)
      (dionysos--process-sentinel process event
                                  (symbol-value G66)))
   '--after-fn-- --cl-rest--))

In the Messages buffer :

[dionysos-process] Next: (lambda nil (message OK))
OK

So I guess my mistake comes from my unit test or my configuration of ert-runner ?
Any idea ?
Thanks.

The ert-runner batch script fails in the following line:

if not [%inside_emacs%]==[] (

because inside_emacs is set to 27.0.90,compile.

Error message is:

"compile]==[]" kann syntaktisch an dieser Stelle nicht verarbeitet werden.

Versions used:

• Cask: 0.8.4
• ert-runner: 20200321.2158
• Emacs: GNU Emacs 27.0.90 (build 1, x86_64-w64-mingw32)
• Windows: Microsoft Windows 10 Pro

I'm trying to convert my package, ido-ubiquitous, to ert-runner. My package cannot function in batch-mode, so presumably I need to use the --no-win or --win option. However, when I do this, an Emacs window pops up for a fraction of a second and then disappears, leaving no output of any kind on the command line. The command exits with code 1. Can you shed any light on what's going wrong here?

ert--print-backtrace has been removed with this commit

emacs-mirror/emacs@e09120d

So ert-runner can't work with latest Emacs. Please help to fix.

Thanks

Hi!
I added some very rudimentary coverage metrics to ert-runner in this branch:

https://github.com/jorgenschaefer/ert-runner.el/tree/coverage

I had a few problems getting this to actually run. The tests seem not to copy the ert-runner-cover.el file, and I didn't find where to add it. There also was some funny problem with the % in the feature test being interpreted as a formatting control, but I couldn't reproduce that anymore.

Also, feature tests do not really lend themselves to testing the ert-runner-coverage.el functions, and I didn't know where (or whether!) to add that. ert-runner self-testing would be great of course :-D

The biggest problem is that the debug coverage information is rudimentary at best. I haven't been able to figure out when it considers a form run or not. For example, defined functions are never "called" as such.

Also, this is simple frequency counting, not branch coverage. Both the coverage testing in edebug as well as in testcover don't do branch coverage, but require every form to return at least two different values to consider it "covered", which I think is bogus.

Now, I do not really see myself having much time to work on this in the near future, but I figured in case someone wants to continue this work they can base it on this framework.