rubyworks / lemon Goto Github PK

View Code? Open in Web Editor NEW

9.0 3.0 3.0 1.73 MB

Pucker-Strength Unit Testing

Home Page: http://rubyworks.github.com/lemon

License: Other

Ruby 100.00%

lemon's Introduction

Lemon

Homepage | User Guide | Development | Issues

DESCRIPTION

Lemon is a Unit Testing Framework that enforces a strict test structure mirroring the class/module and method structure of the target code. Arguably this promotes the proper technique for low-level unit testing and helps ensure good test coverage.

The difference between unit testing and functional testing, and all other forms of testing for that matter, is simply a matter of where the testing concern lies. The concerns of unit testing are the concerns of unit tests, which, in object-oriented design, drills down to individual methods.

IMPORTANT! As of v0.9+ the API has changed. The unit :name => "description" notation is no longer supported.

EXAMPLE

Lemon tests are broken down into target class or module and target methods. Within these lie the actual tests.

Let's say we have a script 'mylib.rb' consisting of the class X:

class X
  def a; "a"; end
end

An test case for the class would be written:

covers 'mylib'

testcase X do

  setup "Description of setup." do
    @x = X.new
  end

  method :a do
    test "method #a does something expected" do
      @x.a.assert.is_a? String
    end

    test "method #a does something else expected" do
      @x.a.assert == "x"
    end
  end

end

The covers method works just like require with the exception that it records the file for reference --under certain scenarios it can be used to improve test coverage analysis.

The setup (also called the concern) is run for every subsequent test until a new setup is defined.

In conjunction with the #setup method, there is a #teardown method which can be used "tidy-up" after each test.

The #unit method is also aliased as #methed which is actually a bit more readable. Along with that there is class_unit and it's alias class_method for testing class-level methods. Also note that the test methods may be capitalized (e.g. `#TestCase'), if you prefer that style.

That is the bulk of the matter for writing Lemon tests. To learn about additional features not mentioned here, check-out the User Guide.

USAGE

Running Tests

Tests can be run using the lemons command line tool.

$ lemons test test/cases/name_case.rb

Lemon utilizes the RubyTest universal test harness to run tests, the lemons test command simply passes off control to rubytest command. So the rubytest command-line utility can also be used directly:

$ rubytest -r lemon test/cases/name_case.rb

Normal output is typically a dot progression. Other output types can be specified by the --format or -f option.

$ rubytest -r lemon -f tapy test/cases/name_case.rb

See RubyTest for more information.

Checking Test Coverage

Lemon can check per-unit test coverage by loading your target system and comparing it to your tests. To do this use the lemons coverage command.

$ lemons coverage -Ilib test/cases/*.rb

The coverage tool provides class/module and method coverage and is meant as a "guidance system" for developers working toward complete test coverage. It is not a LOC (lines of code) coverage tool and should not be considered a substitute for one.

Generating Test Skeletons

Because of the one-to-one correspondence of test case and unit test to class/module and method, Lemon can also generate test scaffolding for previously written code. To do this, use the lemons generate or lemons scaffold command line utilities.

The generate command outputs test skeletons to the console. You can use this output as a simple reference or redirect the output to a file and then copy and paste portions into separate files as desired. The scaffold command will create actual files in your test directory. Other than that, and the options that go with it (e.g. --output), the two commands are the same.

To get a set of test skeletons simply provide the files to be covered.

$ lemons generate lib/**/*.rb

The generator can take into account tests already written, so as not to include units that already have tests. To do this provide a list of test files after a dash (-).

$ lemons generate -Ilib lib/**/*.rb - test/**/*.rb

Test skeletons can be generated on a per-file or per-case bases. By case is the default. Use the -f/--file option to do otherwise.

$ lemon scaffold -f lib/foo.rb

The default output location is test/. You can change this with the -o/--output option.

Generating test case scaffolding from code will undoubtedly strike test-driven developers as a case of putting the cart before the horse. However, it is not unreasonable to argue that high-level, behavior-driven, functional testing frameworks, such as Q.E.D. and Cucumber, are better suited to test-first methodologies. While test-driven development can obviously be done with Lemon, unit-testing best suited to testing specific, critical portions of code, or for achieving full test coverage for mission critical applications.

Test Directory

There is no special directory for Lemon tests. Since they are unit tests, test/ or test/unit/ are good choices. Other options are cases/ and test/cases since each file generally defines a single test case.

COPYRIGHTS

Lemon Unit Testing Framework

Lemon is distributable in accordance with the FreeBSD license.

See the LICENSE.txt for details.

lemon's People

Contributors

Stargazers

Watchers

Forkers

yosefbennywidyo felixonmars sysfce2

lemon's Issues

Reusable Lemons

Instead of Lemon being it's own test framework, consider how it could be made a reusable library with adapter DSLs for other frameworks, in particular TestUnit/MiniTest, QED, KO and Cucumber.

Improvements needed for TAP-Y/J

The TAP-Y/J formats have been added but they are not using all available fields b/c lemon needs to pass more information along to each unit in order to do so.

Before and After without arguments?

Not sure that #before and #after without any match arguments should be the same as #prepare and #cleanup, as they currently are in version 0.8.4.

Flat test list

Currently the DSL collects tests into a tree hierarchy of suite -> case -> unit, and eventually may become suite -> case -> unit -> test (see layer issue). A more elegant solution may be to have a flat list, such that each test object has everything it needs to run on it's own, and a group field to track how it's is group together with other tests. I think this would greatly simplify the underlying code base.

The test object would be constructed something like:

Test.new(
  :module =>
  :method =>
  :singleton =>
  :description =>
  :setup =>
  :teardown =>
  :groups => [ ... ],
  &procedure
)

Ok/No Checks

Currently we have something like this:

TestCase HelloWorld do

  Setup do
    HelloWorld.new
  end

  Unit :hello => "aspect" do |hello_world|
    ...
  end

end

This creates a "setup" and the return value of the setup procedure is passed to the unit test.

One of the features of KO that would be nice to see in Lemon is ok/no checks. These are calls that rerun a test passing
new arguments to it. In KO it look like this:

Test.case HelloWorld do

  test "aspect" do |arg|
     ...
  end

  ok "arg"
  no "arg"

end

We could move Lemon in that direction by simply adding ok/no methods, like so:

TestCase HelloWorld do

  Unit :hello => "aspect" do |arg|
    ...
  end

  Ok "arg"
  No "arg"

end

But then how does setup's return value come into play? Do we simple combine the arguments?

TestCase HelloWorld do

  Setup do
    HelloWorld.new
  end

  Unit :hello => "aspect" do |hello_world, arg|
    ...
  end

  Ok "arg"
  No "arg"

end

First off, it may be very tricky to implment this combination, as currently it depends on arity == 0 or not as to whether setup's return value is passed, which would no longer work.

On second thought, the use of Ok/No style tests will often preclude the use of setup return value b/c the arguments are likely to define a new instance of the target class. Perhaps then setup should take the arg and pass any needed args on the the test method.

TestCase HelloWorld do

  Setup do |arg1, arg2|
    [ HelloWorld.new(arg1), arg2 ]
  end

  Unit :hello => "aspect" do  |hello_world, arg2|
    ...
  end

  Ok "arg1", "arg2"
  No "arg1", "arg2"

end

If there is no setup method, then the args are passed directly to the unit test.

It seems a bit strange but it is clearly the corrent arrangement.

Add additional test layer

For next major version consider adding an additional layer to the DSL, e.g.

testcase Example do

  unit :f do

    test "will do something" do
      ...
    end

    test "will do something else" do
      ...
    end

  end

end

In contrast to the current:

testcase Example do

  unit :f => "will do something" do
    ...
  end

  unit :f => "will do something else" do
    ...
  end

end

Nested Unit Tests

To organize test better, what about using a nest unit design as follows:

TestCase HelloWorld do

  Unit :hello do

    Test "description ..." do
      ...
    end

    Test "description ..." do
      ...
    end

  end

end

This would fit in well with Ok/No style tests too.

TestCase HelloWorld do

  Unit :hello do

    Test "description ..." do |arg|
      ...
    end

    Ok "arg"
    No "arg"

    Test "description ..." do |arg|
      ...
    end

    Ok "arg"
    No "arg"

  end

end

However #Setup does become a little bit trickier. Could it go inside or outside to the unit clause or both?

Missing git tags for 0.8.5, 0.9.0, 0.9.1 releases

It would be nice to keep RubyGems releases and git tags in sync :)

lemon 0.8.3 fails its own tests

I'm trying to run lemon's own tests to verify that it works as expected, but I'm getting an error. I'm using the following invocation:

PATH="bin:${PATH}" RUBYLIB="lib" test/runner

which results in the following error:

ERROR: undefined method `ansi' for "Lemon::CoverageAnalyzer":String

Not catching methods called without receiver?

Lemon doesn't seem to catch methods called from a private context (or some such reason). Eg.

Covers 'facets/module/abstract'

Case Module do

  MetaUnit :abstract => "in an anonymous class" do
    c = Class.new{ abstract :q }
    x = c.new
    TypeError.assert.raised?{ x.q }
  end

end

Doesn't work. It never sees that #abstract is called. But if we do:

c.pry.abstract :q

It does work.

Add `ruby2_keywords` for argument forwarding.

The lemon gem needs an update to support Ruby 3.x - without it I can't update the tests for facets.

You need to add ruby2_keywords to the argument forwarding:

https://github.com/rubyworks/lemon/blob/master/lib/lemon/test_method.rb#L86-L89

Otherwise keyword arguments won't be forwarded correctly.

lemon does not return a shell exit code

I would expect lemon to return an exit code other than 0 when errors are encountered, but it seems to always return 0. This makes it impossible to integrate lemon into any kind of automated mechanism for running tests since the failure condition isn't clear.

Instance/Singleton for Setup

In a previous version, the Setup/Concern methods could also be taken up by the Instance or Singleton methods. These worked the same way except that they would determine if the subsequent unit tests were instance methods tests or class-method tests, respectively. For example:

TestCase Foo do

  Singleton "some description"

  Unit :some_class_method do
    ...
  end

  Instance "some description" do
    Foo.new
  end

  Unit :some_instance_method do
    ...
  end

end

This seem like a great idea originally as it made the Meta/MetaUnit method moot. But it does mean the unit tests are even more context sensitive than before. In fact, b/c of that, it seems almost more prudent to have a alternative #TestCase method.

TestCase Foo.singleton_class do

  Unit :some_class_method do
    ...
  end

end

TestCase Foo do

  Setup "some description" do
    Foo.new
  end

  Unit :some_instance_method do
    ...
  end

end

lemon 0.9.0 fails tests

Trying to run lemon 0.9.0's test suite with 'rake test' I get the following error in ruby-test:

/usr/lib64/ruby/gems/1.8/gems/test-0.2.1/lib/test/rake.rb:85:in `run': undefined method `new' for #<Test::Rake::TestTask:0x7f4624826fb0> (NoMethodError)
    from /usr/lib64/ruby/gems/1.8/gems/test-0.2.1/lib/test/rake.rb:76:in `fork'
    from /usr/lib64/ruby/gems/1.8/gems/test-0.2.1/lib/test/rake.rb:76:in `run'
    from /usr/lib64/ruby/gems/1.8/gems/test-0.2.1/lib/test/rake.rb:70:in `define_task'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
    from /usr/lib64/ruby/1.8/monitor.rb:242:in `synchronize'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
    from /usr/lib64/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31
    from /usr/bin/rake:8:in `load'
    from /usr/bin/rake:8

lemon 0.9.0 refers to tests that do not exist

lemon 0.9.0's Rakefile refers to a test directory with tests, but this directory no longer exists:

  desc 'run lemon unit tests (via shell command)'
  task :unit do
    sh 'lemonade test -Itest/fixtures test/*.rb'
  end