ruby / mspec Goto Github PK

With Appveyor jobs being run for both mswin & mingw, people occasionally bypass testing using:

platform_is_not :windows

but the tests are only failing on one or the other, but not both. Are there guards for mswin and mingw?

I don't know whether they should be options for platform_is_not or platform_is, but mswin and mingw are part of RUBY_PLATFORM...

I am thinking to only keep one config file named ruby.mspec since they are currently all identical.
And then specify --config ruby.mspec to mspec.
What do you think?

we have needless branches without master. I will delete these branches after pass rubyci

This would allow to report errors as they are observed and be more flexible.
Currently, errors are delayed until the end as they are dumped into a YAML file.
This will need an improved protocol between the master mspec and the sub-processes.

https://travis-ci.org/ruby/ruby/builds/160663397 failed at make test-rubyspec MSPECOPT=-fm.

$ make test-rubyspec MSPECOPT=-fm
generating x86_64-linux-fake.rb
x86_64-linux-fake.rb updated
./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems -r./x86_64-linux-fake ./spec/mspec/bin/mspec run -B ./spec/default.mspec -fm
ruby 2.4.0dev (2016-09-17 trunk 56179) [x86_64-linux]
/home/travis/build/ruby/ruby/spec/mspec/lib/mspec/utils/options.rb:290:in `block in formatters': uninitialized constant MSpecOptions::SummaryFormatter (NameError)
    from /home/travis/build/ruby/ruby/spec/mspec/lib/mspec/utils/options.rb:107:in `process'
    from /home/travis/build/ruby/ruby/spec/mspec/lib/mspec/utils/options.rb:147:in `parse'
    from /home/travis/build/ruby/ruby/spec/mspec/lib/mspec/commands/mspec-run.rb:77:in `options'
    from /home/travis/build/ruby/ruby/spec/mspec/lib/mspec/utils/script.rb:224:in `main'
    from /home/travis/build/ruby/ruby/spec/mspec/bin/mspec-run:7:in `<main>'
make: *** [test-rubyspec] Error 1

Currently, combining guards is complicated, and is often side-stepped by using shared specs which introduce an extra indirection.

Going forward I would like that guards also support being called without a block and just return true/false. Then with some generic guard/guard_not taking a lambda for the guard we could do just like if/unless, but also be able to ignore all guards (--unguarded). I am currently pondering whether --unguarded is really useful. If not, we could remove it and then just use something like:

if platform_is :linux or (platform_is :windows and ruby_version_is "2.5")
  # specs
end

which I believe is much simpler, but prevents ignoring guards globally (--unguarded).

@eregon

I'm re-writing ruby-loco, similar to what I helped with in Core. I've got a few spec patches that remove guards placed around tests for mingw.

But, I've only been testing with trunk, I'm now locally checking them with older Ruby versions. I need a guard to skip a test like is :mingw and RUBY_VERSION < '2.4.0'.

Any suggestions or prefererence?

Re the Ruby version guards, it would be helpful if there was something like less_than and greater_than (or greater_than_or equal). Is there something similar to that?

Lastly, I need to spend more time looking at all the options in mspec. Is there a way to output all skipped tests? In testing, the numbers jump around a little, but some may be due to parallel testing. BTW, things got a bit squirrely with --random...

Thanks, Greg

Currently RSpec 2 is used but this is really not ideal as it is very easy to accidentally override some RSpec methods by require-ing the full mspec. It's also confusing as both frameworks use mostly the same methods.

Tests for mspec itself should be written in a very simple test harness like minitest, or for instance mruby's assert.rb, so that newer implementations can easily run them.

The current tests are heavily relying on mocks, so that might be non-trivial to replace.
In general I'm in favor of removing these white-box mock tests and instead test at a higher level for the functionality and not for a specific implementation.

While implementing shared specs for ruby/spec#576, I noticed some limitations in the implementation of it_behaves_like that should be addressed.

Issues Discovered

1. All it_behaves_like calls must have an identical @method and @object calls within a given scope, or they will affect each others' state. To overcome this, you have to wrap the differing parameter in a context or describe block.

2. If you have a shared spec that relies upon another shared spec, the above issue gets more complex, because even if you've wrapped it in a describe block, the @method has shared scope. To get around this, I have saved the outer @method into a new @base_method inside a before and referred to it via @base_method to solve the collision.

Proposed Solution

Instead of mapping to @method and @object, scoping these to instance variables specific to the behaves_like description:

def it_behaves_like desc, meth, obj=nil
  send :before, :all do
    @shared_bindings ||= {}
    @shared_bindings[desc] = { method: meth, object: obj }
  end
end

I haven't fully fleshed this out, but if it sounds like a reasonable approach I can go further on a fleshed out solution and PR accordingly.

Dear all.

I'm working on project what run specs ruby/spec on mruby.

https://github.com/ksss/mruby-spec

I have some feedbacks to run specs ruby/spec on mruby.

1. "str" "str" is SyntaxError
2. mruby doesn't have defined?
3. require 'rbconfig'
4. require 'pp'
5. require 'iconv'
6. Need some patch

See also mruby-spec.

I can send a PR from my forked mspec to here.

This PR can fix 1. and 2.
But 3. ~ 6. is not.

3. is using as hooks of 6. because require 'rbconfig' is called by beginning part.

4. can avoid to use mruby-pp and make a empty file for require.

5. problem is still undecided. because I don't encount this issue. But this issue will be able to cope by mruby-iconv

6. I have no idea. I can add some patch to here. But I think it will become issue of each library if we want to resolve, probably.

On Solaris 10, the following error frequently occurs.

  -e:1: `$(' is not allowed as a global variable name

This error occurs in the ruby_cmd() method.
In lib/mspec/helpers/ruby_exe.rb line 152,

        body = %Q!-e "$(cat <<'#{heredoc_separator}'\n#{code}\n#{heredoc_separator}\n)"!

In the line, $() which should be interpreted by the system's shell is used.
However, on Solaris, because /bin/sh is traditional /bin/sh, $() can not be interpreted.
I think $() is a bash extension, and I think the error can occurr other OSs whose /bin/sh is not bash.

We have to fix it:

Failures:
  1) BlockingMatcher does not match when a Proc does not block the caller
     Failure/Error: BlockingMatcher.new.matches?(proc { 1 }).should == false
       expected: false
            got: true (using ==)
     # ./spec/matchers/block_caller_spec.rb:11:in `block (2 levels) in <top (required)>'

The documentation link in the README currently points to rubyspec.org. Docs are no longer available there; the link should be updated to point to an updated source of documentation for mspec.

Currently, the MultiFormatter always uses the SpinnerFormatter but this is not always appropriate (e.g.: CI).

After d0406c9, the followinf errors occur.

./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disa
ble-gems -r./sparc64-solaris2.10-fake ./spec/mspec/bin/mspec run -B ./spec/default.mspec 
ruby 2.3.0dev (2015-09-02) [sparc64-solaris2.10]
...................
(snip)
sh: END_OF_RUBYCODE: not found
sh:  2>&1: not found
(snip)
F...............................................................................
................................................................................
................................................................................
......................................................E.........................
................................................................................
................................................................................
...............................................................................F
...............................F....FF.............F............................
................................................................................
................................................................................
................................................................................
.................Broken Pipe
.....................................sh: : cannot execute
Fsh: : cannot execute
F...................sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
sh: : cannot execute
F...............................................................................
(snip)
11)
Kernel#autoload when Object is frozen raises a RuntimeError before defining the 
constant FAILED
Expected "-e:5: syntax error, unexpected end-of-input\n"
to equal "RuntimeError - nil"

/XXXXX/spec/rubyspec/core/kernel/autoload_spec.rb:75:in `block (3 levels) in <top (required)>'
/XXXXX/spec/rubyspec/core/kernel/autoload_spec.rb:16:in `<top (required)>'
(snip)

The error messages "sh: END_OF_RUBYCODE: not found" and "sh: 2>&1: not found" are caused by /bin/sh in Solaris that may parse the double quote " in cat contents and the command-line is wrongly splitted by the shell.

I think using here document simply like below might solve the problem.

ruby - <<'HERE_DOCUMENT'
puts 0
HERE_DOCUMENT

I also thing that using temporary script file may be an alternative solution.

$ MSPECOPT="--verbose" make test-rubyspec
(snip)
/XXXXX/trunk-51741/spec/rubyspec/language/predefined/data_spec.rb                                           /XXXXX/trunk-51741/spec/rubyspec/language/predefined/fixtures/data_flock.rb:1:in `flock': Bad file number @ rb_file_flock - /XXXXX/trunk-51741/spec/rubyspec/language/predefined/fixtures/data_flock.rb (Errno::EBADF)
        from /XXXXX/trunk-51741/spec/rubyspec/language/predefined/fixtures/data_flock.rb:1:in `<main>'
F
(snip)

1)
The DATA constant succeeds in locking the file DATA came from FAILED
Expected ""
 to equal "0"

/XXXXX/trunk-51741/spec/rubyspec/language/predefined/data_spec.rb:31:in `block (3 levels) in <top (required)>'
/XXXXX/trunk-51741/spec/rubyspec/language/predefined/data_spec.rb:3:in `<top (required)>'
(snip)

On Solaris, write access is needed to lock a file.
This is well known difference of behavior between Solaris and Linux.
Plese see previous discussions in http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/291535

At this point ruby/mspec has diverged just enough from rubinius/mspec (originally rubyspec/mspec) that the released mspec gem (based on rubinius/mspec) can't be used to run ruby/rubyspec anymore. Primarily, the addition of the "argf" helper prevents the specs from running green:

...
14)
ARGF.lines is a public method ERROR
NoMethodError: undefined method `argf' for #<Object:0xcc81cb3>
/Users/headius/projects/jruby/spec/ruby/core/argf/shared/each_line.rb:11:in `block in (root)'
org/jruby/RubyBasicObject.java:1633:in `instance_eval'
org/jruby/RubyEnumerable.java:1554:in `all?'
org/jruby/RubyFixnum.java:296:in `times'
org/jruby/RubyArray.java:1560:in `each'
/Users/headius/projects/jruby/spec/ruby/core/argf/lines_spec.rb:4:in `<top>'
org/jruby/RubyKernel.java:957:in `load'
org/jruby/RubyBasicObject.java:1633:in `instance_eval'
org/jruby/RubyArray.java:1560:in `each'
/Users/headius/projects/jruby/lib/ruby/gems/shared/gems/mspec-1.9.1/bin/mspec-ci:8:in `<top>'

15)
ARGF.lines requires multiple arguments ERROR
NoMethodError: undefined method `argf' for #<Object:0xcc81cb3>
/Users/headius/projects/jruby/spec/ruby/core/argf/shared/each_line.rb:17:in `block in (root)'
org/jruby/RubyBasicObject.java:1633:in `instance_eval'
org/jruby/RubyEnumerable.java:1554:in `all?'
org/jruby/RubyFixnum.java:296:in `times'
org/jruby/RubyArray.java:1560:in `each'
/Users/headius/projects/jruby/spec/ruby/core/argf/lines_spec.rb:4:in `<top>'
org/jruby/RubyKernel.java:957:in `load'
org/jruby/RubyBasicObject.java:1633:in `instance_eval'
org/jruby/RubyArray.java:1560:in `each'
/Users/headius/projects/jruby/lib/ruby/gems/shared/gems/mspec-1.9.1/bin/mspec-ci:8:in `<top>'
...

I would like to be able to set up JRuby's dev process to use a released gem rather than a vendored copy of mspec. I see a few courses of action possible:

1. Submit the argf modification to rubinius/mspec. This would allow using the mspec gem based on that repository to run ruby/rubyspec. However, it is a temporary measure as long as the ruby organization maintains a fork of mspec. There's also some chance the change would be rejected, since it is not needed by rubinius/rubyspec.
2. Release ruby/mspec as a new gem ruby-mspec (or similar name). This may lead to confusion with the existing mspec gem, but I don't think there's many projects using it.
3. Revert the argf change and return to using rubinius/mspec. This would get us back on the same page, but it limits what changes we can make to improve mspec for CRuby, JRuby, and other non-Rubinius implementations' use of it and ruby/rubyspec.

@eregon

When trying to get make test-spec work parallel, Windows / MinGW has an issue with temp files, often when they're used in a block. Seems to be due to a process not closing all files before ending.

This generates the error in lib/mspec/helpers/tmp.rb at_exit.

I added the following patch file, tests report the same, CAPI *.so files are created, etc. I tested both using make test-spec and mspec.

I'm just not sure if this allows other possible failures/errors/skips to not be reported...

diff --git a/spec/mspec/lib/mspec/commands/mspec.rb b/spec/mspec/lib/mspec/commands/mspec.rb
index 6b4354c2b0..5f90647858 100755
--- a/spec/mspec/lib/mspec/commands/mspec.rb
+++ b/spec/mspec/lib/mspec/commands/mspec.rb
@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 #!/usr/bin/env ruby

 require 'mspec/version'
@@ -7,6 +8,7 @@
 require 'mspec/runner/actions/filter'
 require 'mspec/runner/actions/timer'

+SPEC_WIN = /mswin|mingw/ =~ RUBY_PLATFORM

 class MSpecMain < MSpecScript
   def initialize
@@ -100,7 +102,7 @@ def multi_exec(argv)
     output_files = []
     processes = cores(@files.size)
     children = processes.times.map { |i|
-      name = tmp "mspec-multi-#{i}"
+      name = tmp String.new("mspec-multi-#{i}")
       output_files << name

       env = {
@@ -127,7 +129,7 @@ def multi_exec(argv)
           while chunk = (io.read_nonblock(4096) rescue nil)
             reply += chunk
           end
-          raise reply
+          raise reply unless SPEC_WIN
         end
         io.puts @files.shift unless @files.empty?
       }

Currently ruby_bug only supports <= ceil(VERSION), but having a lower bound might be useful if it was actually introduced as a regression.
I will have a try when things get stable.

When Mock.install_method aliases an existing method (https://github.com/ruby/mspec/blob/master/lib/mspec/mocks/mock.rb#L63), it does so using a symbol that is unique (it has the #object_id). And since that symbol is used as method, even if the method gets erased, the symbol remains.

This means that if a spec was to run multiple time (ex, using repeat), or some custom made mechanism, specs using Mock.install_method appear to have a memory leak (They technically do).

If those names were instead reused, such as by incrementing from 1, storing the number, and then on clenaup, resetting to 1, the leaky side of this would be removed.

Thoughts?