Not sure if to raise this here or https://github.com/flatiron-labs/learn-co/ but when running learn to run the test suite of a lesson, the first thing it does is authenticate, which is slow, is prone to environmental failure, and delays feedback of test results.

When learn is evoked and determined to run a lesson's test suite, the priority should be providing the student with meaningful output about the test suite/run. Ideally running learn should be as quick as running rspec to the user. Currently, running learn adds a significant delay in the feedback cylce mostly do to authenticating and submitting the test results to Learn.

Suggestions

1. Re-order procedure of learn-test to first run test suite and then authenticate and submit.

learn should immediately try to run the test suite for the user and provide output. Upon a run of the test suite, assuming we can rescue complete build/interpreter/compilation failures. we should only then connect to learn and submit the results. But by then, the learner has seen the test suite's output and has even, ideally, be dropped back into an active prompt while the submission of the test runs occur entirely behind the scenes.

1. Optimize Authentication so it isn't so slow

Through caching or assuming the best from the user (they aren't trying to run it for someone else's credit), do we really need to authenticate every time? We can just assume they are correctly logged in, submit the post with a cached token, then run the test suite. If the POST fails, we can ask them to re-login or trouble shoot.

We should make sure that all of the dependences listed in the package.json are present in node_modules.

It'd be cool if the CLI could do a verification of your ENV and report any warnings or issues.

$ learn doctor
Checking your environment...
...gcc ok
...non apple ruby ok

etc

Add support for --bail flag for mocha tests

Proposed usage(s):

• learn --bail
• learn -b
• learn test --bail
• learn test -b

Expected behavior

See mocha command line usage
'Abort ("bail") after first test failure'

Mimics usage of --fail-fast

On WSL 2, the learn command runs tests successfully, but then consistently throws the following error when it's done, requiring a force interrupt:

Rolling back to 3.1.2 works as a temporary fix, but since 3.2.0 seems to introduce new functionality that's throwing the error, I thought you might want to be aware of it.

Please see this issue:
learn-co/learn-ide#428

@kvignali commented on Wed May 17 2017

https://www.screencast.com/t/JVDyLHDMc

@drewprice commented on Wed May 17 2017

It actually looks like something else is making this happen. The test is returning, which brings the prompt back, but then something is being written to stdout after the fact—usually something like this would be caused by a background process of some sort.

Not positive, but I think learn-test-sender (which is what's throwing the second error that makes it look all broken) might be related to the experiment we've been running with the learn-test gem.

We should look into this, but it seems very unlikely that this is related to this IDE release.

Runs local test suite in current directory

Just learn is alias for this

This leaves the learn test suite hanging and requiring a system interrupt which leads to the tests not posting to learn-co.

Mocha changed this functionality in Mocha 4.0.0 and leads to numerous amounts of labs not able to submit tests.
Read more on issues related to Mocha tests hanging here.

Fixing this only requires us to add a new flag that Mocha defined as --exit. Which will close the mocha terminal and allow us to submit our completed tests.

I also made another issue on this for one of Flat Iron Schools labs which you can view here. This was somewhat manageable when mocha multi was not needed by manually adding the flag to our npm test script defined in package.json to include the --exit tag.

Although when mocha multi is needed it uses the script defined within mocha.rb on line 74.

If somebody runs learn on a lab that has rspec in the Gemfile, but hasn't bundled yet, it fails with no output. Let's get some output that mentions running bundle install.

learn --fail-fast should mimic rspec --fail-fast but these flags don't work right now. Also, when this is added back, documentation for it should be added to learn help test.
If students want to only see one spec at a time, they now use rspec --fail-fast which will end up leaving heartbeat with less data.

using Windows 10 CMD (will not work from power shell at all)

Steps to reproduce:

1. run the command learn-test after it is installed on your system
2. be sure you are not logged in

stack-trace (after tests have run):

Enter your github username: C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/username_parser.rb:10:in `gets': No such file or directory @ rb_sysopen - --format documentation (Errno::ENOENT)
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/username_parser.rb:10:in `gets'
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/username_parser.rb:10:in `get_username'
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/strategy.rb:40:in `username'
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/strategies/rspec.rb:45:in `results'
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/runner.rb:72:in `push_results'
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/lib/learn_test/runner.rb:19:in `run'
        from C:/RailsInstaller/Ruby2.3.0/lib/ruby/gems/2.3.0/gems/learn-test-2.4.2/bin/learn-test:60:in `<top (required)>'
        from C:/RailsInstaller/Ruby2.3.0/bin/learn-test:22:in `load'
        from C:/RailsInstaller/Ruby2.3.0/bin/learn-test:22:in `<main>'

It'd be awesome to be able to send a --local flag to learn-test so that it just wraps the appropriate test runner and doesn't try submitting or authenticating. Would make running 'test' runs way faster.

Relates to #7

The regex on this line is expecting https, but if the remote is set to http for whatever reason, it errors - https://github.com/learn-co/learn-test/blob/master/lib/learn_test/repo_parser.rb#L14

error:

/usr/local/opt/rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/learn-test-2.0.0/lib/learn_test/repo_parser.rb:14:in `get_repo': undefined method `[]' for nil:NilClass (NoMethodError)
        from /usr/local/opt/rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/learn-test-2.0.0/bin/learn-test:55:in `<top (required)>'
        from /usr/local/opt/rbenv/versions/2.1.5/bin/learn-test:23:in `load'
        from /usr/local/opt/rbenv/versions/2.1.5/bin/learn-test:23:in `<main>'
        from /usr/local/opt/rbenv/versions/2.1.5/bin/ruby_executable_hooks:15:in `eval'
        from /usr/local/opt/rbenv/versions/2.1.5/bin/ruby_executable_hooks:15:in `<main>'```