Giter Club home page Giter Club logo

tapout's Introduction

TAPOUT

Website | User Guide | Report Issue | Development

Build Status Gem Version     Flattr Me

About

TAPOUT is the next generation in test results viewing. You may have heard of Turn or minitest-reporters. TAPOUT is the conceptual successor to these gems by virtue of its use of TAP, a standardized intermediate test results protocol.

TAPOUT works as a TAP handler which supports TAP-Y/J as well as traditional TAP streams. TAP-Y/J is a modernization of TAP using pure YAML/JSON streams. Traditional TAP has less detail than TAP-Y/J, but it can still be translated with fairly good results. TAPOUT includes a TAP adapter to handle the translation transparently. Currently TAPOUT supports TAP v12 with some minor limitations.

To learn about the TAP-Y/J specification, see the TAP-Y/J Specification document.

For information about TAP, see http://testanything.org/

Usage

To learn more about using Tapout, please see the wiki. It provides more detailed information on how to put Tapout to work for you using your preferred testing framework and build tool. What follows here is a very general overview of usage.

To use TAPOUT you need either a plugin for your current test framework, or use of a test framework that supports TAP-Y/J out of the box. You can find a list of plugins here under the section "Producers".

With a test framework that produces a TAP-Y/J output stream in hand pipe the output stream into the tapout command by using a standard command line pipe.

$ rubytest -y -Ilib test/foo.rb | tapout

TAPOUT supports a variety of output formats. The default is the common dot-progress format (simply called dot). Other formats are selectable via the tapout command's first argument.

$ rubytest -y -Ilib test/foo.rb | tapout progressbar

TAPOUT is smart enough to match the closest matching format name. So, for example, the above could be written as:

$ rubytest -y -Ilib test/foo.rb | tapout pro

And tapout will know to use the progressbar format.

To see a list of supported formats use the list subcommand:

$ tapout --help

If your test framework does not support TAP-Y/J, but does support traditional TAP, TAPOUT will automatically recognize the difference by TAP's 1..N header and automatically translate it.

$ rubytest -ftap -Ilib test/foo.rb | tapout progressbar

Bypassing

Since tapout receives test results via a pipe, it has no direct control over the producer, i.e the test runner. If you need to tell tapout to stop processing the output then you can send a PAUSE DOCUMENT code. Likewise you can restart processing by sending a RESUME DOCUMENT code. These codes are taken from ASCII codes for DLE (Data Link Escape) and ETB (End of Transmission Block), respectively. When tapout receives a PAUSE DOCUMENT code, it stops interpreting any data it receives as test results and instead just routes $stdin back to $stdout unmodified.

A good example of this is debugging with Pry using binding.pry.

def test_something
  STDOUT.puts 16.chr  # tells tapout to pause processing
  binding.pry
  STDOUT.puts 23.char # tells tapout to start again
  assert something
end

As it turns out, if your are using TAP-Y (not TAP-J) then you can also use YAML's END DOCUMENT marker to acheive a similar result.

def test_something
  STDOUT.puts "..."  # tells tapout to pause processing
  binding.pry
  assert something
end

But this only works for YAML and if you happened to be debugging code that emits YAML you might accidentally trigger tapout to resume. Therefore it is recommended that the ASCII codes be used.

Note: When sending these codes, be sure to send a newline character as well.

Legal

Copyright (c) 2010 Rubyworks

TAPOUT is modifiable and redistributable in accordance with the BSD-2-Clause license.

See COPYING.md for details.

tapout's People

Contributors

trans avatar dylandavidson avatar leppert avatar seanhussey avatar doabit avatar

Stargazers

Benjamin P. avatar Phil Pirozhkov avatar Juri Hahn avatar Alexandre Joannou avatar Adam S. avatar Ryan Moore avatar Thomas Vendetta avatar Martin Kopischke avatar Terje Larsen avatar Lorenzo Bragaglia avatar Yuriy Pitomets avatar Mike Chau avatar Christoph Grabo avatar Mozart Brum avatar Paul Keen avatar Michael Mahemoff avatar asymmetric avatar Alex Riedler avatar Magnolia.K avatar Johnson Denen avatar Sergey Moiseev avatar Michael D.W. Prendergast avatar Martin Lagrange avatar Volkan Unsal avatar Steve Loveless avatar Lewis Brown avatar Anatoly Chernov avatar Robert Berry avatar Michael Yared avatar Clint Plummer avatar Yuan Heng Fan avatar Jin Lee avatar Dominik Richter avatar Abinoam P. Marques Jr. avatar Joshua C Elliott avatar Vojtěch Kusý avatar Caige Nichols avatar Matt Button avatar Mohammad Sadegh Khoeini avatar Nicklas Gummesson avatar Jeremy Jackson avatar Phillip Reichelt avatar Chris Olstrom avatar Alexander Kirillov avatar Vlad Bokov avatar  avatar Oleg Pudeyev avatar Rick Benavidez avatar Val Packett avatar Mike Pack avatar Denys Slipetskyy avatar  avatar Mathias Lafeldt avatar

Watchers

 avatar Abinoam P. Marques Jr. avatar James Cloos avatar  avatar

tapout's Issues

tap reporter with skipped test reports wrong number of tests

The tap reporter does not count or print any skipped tests, although they are accounted for in the test number. E.g. first line prints "1..59", but will only print 56 oks, since 3 tests are skipped.

Here an example by using https://github.com/sass/sass-spec (needs sassc installed)

ruby sass-spec.rb -r spec/basic --impl libsass -V 3.5 -c sassc --tap | tapout tap
1..59
ok 1 - test__sass-spec/spec/basic/33_ambiguous_imports
...
ok 56 - test__sass-spec/spec/basic/57_function_exists

I get it to pass correctly by patching tap_reporter.rb:

    def todo(entry)
      super(entry)
      @i += 1
      puts "ok #{@i} # SKIP #{entry['label']}"
    end

On another note tapout does not seem to distinguish todo and skipped tests, which is IMO not correct according to https://testanything.org/tap-specification.html. Todo tests are expected to fail but should eventually pass (they are considered bugs) and skipped tests might be skipped because they e.g. don't apply to the current version being tested against or are only meant for certain operating systems. Edit: seems the underlying minitest framework does not have the concept of todo tests, so it's probably not relevant.

tapout not defaulting output with minitap and minitest

Using minitap 0.3.0 and tapout 0.3.0 together, the command
rake -q test | tapout
results in a stacktrace

/home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/gems/tapout-0.3.0/lib/tapout/parsers/yaml.rb:13:in `initialize': undefined method `new' for nil:NilClass (NoMethodError)
    from /home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/gems/tapout-0.3.0/lib/tapout.rb:74:in `new'
    from /home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/gems/tapout-0.3.0/lib/tapout.rb:74:in `cli'
    from /home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/gems/tapout-0.3.0/bin/tapout:3:in `<top (required)>'
    from /home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/bin/tapout:19:in `load'
    from /home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/bin/tapout:19:in `<main>'
/home/wyaeld/.rvm/gems/ruby-1.9.2-p180@rails31/gems/minitest-2.6.1/lib/minitest/unit.rb:750:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)

This goes away if you actually specify a report style to output. It seems the default dotprogress selection is getting lost somehow.

Ignoring switches?

Hi again,

The only switch I can get to work is --no-color

The ones I really want to work are trace, lines, and minimal, because I'm doing sooo much scrolling testing my buggy code :)

Any better way to make rake test run through tapout?

The way I could come out is to create a file lib/tasks/tapout.rake with

desc "Run 'rake test' through tapout"
task :tapout, [:reporter] do |t, args|
  reporter = args.reporter || "runtime"
  sh "TESTOPTS='- --tapy' bin/rake -q test | tapout #{reporter}" rescue true
end

So I can run

rake tapout
or
rake tapout[outline]

It's a little weird to make a Rake task to run rake again! Isn't it?

Is there a better way to do it?
If not, could add this (or any improved one) piece of code as example on the Quick Start guide.
I'm asking this because the directions on the Quick Start guide was not working for me (The pipe '|' was not working when inside TESTOPTS).

Running:

minitap (0.5.3)
minitest (5.3.1)
minitest-reporter-api (0.0.5)
rails (4.1.0.rc1)
tapout (0.4.5)

Websocket support

Presently tapout works via a traditional unix pipe. I wonder if also work via websocket would be useful --or other message protocol for that matter-- would be useful.

Imagine a test runner sending TAP-Y/J stream to a service, which in turn sends it out to all listening clients.

Stop/Restart Control Sequence

A user (@doabit) mentioned adding binding.pry to their tests, which drops one into a debugging console at that point in the code. But this can't be used while piping TAP-Y/J output to tapout b/c tapout will try to interpret Pry's interface as test output. Obviously that won't work. It's not the end of the world, one can just run the tests without piping to tapout and it should work fine, but we loose the pretty output while doing that.

So is there a way to have our cake and eat it too? It might be possible for tapout to recognize a special control sequence that instructs it to stop interpreting output as TAP-Y/J and just reroute everything back through stdout/stderr/stdin, until a restart control sequence is seen.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.