The stack trace doesn't have anything to do with the actual test failure:

===== Test Case Failure =====
Location: /home/tgs/multi-lib/data-analysis/cevent/cevent_relative_intervals.m
Name:     cevent_relative_intervals

/home/tgs/src/xunit-git/xunit/DocTest.m at line 203
/home/tgs/src/xunit-git/xunit/DocTest.m at line 100
/home/tgs/src/xunit-git/xunit/TestSuite.m at line 85
/home/tgs/src/xunit-git/xunit/TestSuite.m at line 85
/home/tgs/src/xunit-git/xunit/TestSuite.m at line 85
/home/tgs/src/xunit-git/xunit/runtests.m at line 126

DocTest #4 from cevent_relative_intervals failed; expected first, got second
Inputs are not equal.

First input:
ans = 1 10 1 21 21 2 81 85 3

Second input:
??? Error using ==> cevent_relative_intervals at 76 The WHENCE argument must be either 'start' or 'end'.

So you can just see one function at a time, with nice verbose output and stuff.

doctest some_function

I think that the new subfunction test suites are incompatible with versions of MATLAB before R2013b, since that's when the localfunctions function was added in MATLAB.

I think this is a particular problem since people who know that they're only using R2013b or newer are likely to be using matlab.unittest instead.

If you run your examples in Matlab that's like 130 columns wide, but the tests are run in an environment that's only 80 columns wide, there will be differences in how Matlab paginates the output, so the tests will fail. This only happens if you e.g. print a 30x30 matrix.

In TestSuite.fromName, isdir() is used to determine if the string argument "name" is a subdirectory (TestSuite.m:199). If isdir(name) returns true, xUnit calls cd(name) in TestSuiteInDir.gatherTestCases() (TestSuiteInDir.m:45). The problem is that Matlab knows three types of paths (absolute/full paths, relative paths, partial paths [1]) and isdir() accepts all three of them although this is not explicitly said in the documentation. This may cause an exception when cd() is called in gatherTestCases() because cd() handles only absolute and relative paths (the documentation is clear here). This causes XUnit to throw an exception whenever the name argument to TestSuite.fromName() happens to be a partial path.

Demo:

• Create an empty directory
• Change into the empty directory
• Start Matlab in this directory
• Execute isdir('elmat')
• Execute cd('elmat')

isdir('elmat') will return logical one because MATLABROOT/toolbox/matlab/elmat is in the search path hence "elmat" is a partial path. cd('elmat') will throw an exception because "elmat" is neither a relative nor an absolute path.

Moreover isdir() is called before ispackage() in TestSuite.fromName() so if a package name happens to be a partial path, then an exception will be thrown as well.

[1] http://www.mathworks.com/help/matlab/matlab_env/specify-file-names.html

the testDocTestsHere function needs to give a warning or error if this is the case

also, when constructing the DocTestSuite, it needs to use the full path rather than just the name, because names may exist in multiple places

Hard to know how to run the correct copy of the function if there's more than one, though.

In the main help page (./doc/xunit_product_page.html) the link to the documentation for addpath is broken.

Current link: http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_env/f10-26235.html
Corrected link: https://www.mathworks.com/help/matlab/ref/addpath.html

I realize this package is targeted at older MATLAB releases before the introduction of the matlab.unittest framework, but I wanted to report that matlab-xunit doesn't work anymore in the latest R2016b.

The problem is with function-based unit-tests and the initTestSuite.m script. Unfortunately this hack of running a script from another function won't work to get access to its local functions.

Let me give a simplified example to illustrate the issue.
Consider the following file containing function-based tests implemented as sub-functions:

my_tests.m

function funcs = my_tests
    my_init;

function test_1
    assert(true);

function test_2
    assert(true);

Then consider this simplified version of the init script:

my_init.m

funcs = cell(2,1);
funcs{1} = str2func('test_1');
funcs{2} = str2func('test_2');

Running the file in R2016b we get:

>> f = my_tests
f =
  2×1 cell array
    @test_1
    @test_2
>> functions(f{1})
ans =
  struct with fields:

    function: 'test_1'
        type: 'simple'
        file: ''
>> feval(f{1})
Undefined function or variable 'test_1'. 

So you can see that the function handles returned by str2func are not actually bound to the subfunctions, thus calling them will fail.

To add one more problem, if you call nargout from inside the script (similar to initTestSuite.m), it will actually throw an error now in the latest version.

The only solution I can think of is to use localfunctions introduced in R2013b, which is what matlab.unittest framework relies on:

function funcs = my_tests
    if verLessThan('matlab','9.1')
        my_init;
    else
        funcs = localfunctions;
    end

Now I get this:

>> functions(f{1})
ans = 
  struct with fields:

     function: 'test_1'
         type: 'scopedfunction'
         file: 'C:\path\to\my_tests.m'
    parentage: {'test_1'  'my_tests'}

Of course for matlab-xunit, you'll have to use create a helper function to iterate through the cell array of function handles returned by localfunctions and create a test suite out of them (similar to the related functiontests function but specifically for this package tests), i.e something like this:

function test_suite = test_something
    if verLessThan('matlab','9.1')
        initTestSuite;
    else
        test_suite = initTestSuiteFromFcnHandles(localfunctions);
    end

where initTestSuiteFromFcnHandles will basically filter function handles (extract setup, teardown, and test functions based on their names), then create a TestSuite from those composed of FunctionHandleTestCase tests.

I'm not sure if this issue is restricted to R2016b, so it might be a good idea to test on R2016a and R2015b too, which is around the time MATLAB switched to the new LXE execution engine, which might explain why it broke matlab-xunit..

The documentation for filemarker will be helpful here: http://www.mathworks.com/help/techdoc/ref/filemarker.html

There's some that's just \n and some that are \r\n

Git seems to really want them to be \n in the repository, but since so many Matlab people use it on Windows, I'm not sure if that's best.

There should be a link to some surviving xUnit documentation, but the current link is broken.

The command in the Readme.md file (runxunit ./tests) only runs the tests that are directly in the tests directory and not those in the sub-directories, especially ./tests/src/ etc.