Need to run tests in both strict mode and non-strict mode about test262 HOT 19 CLOSED

(@erights, just fyi)

Investigated this a bit today. Unfortunately this is not easy as many test cases are noStrict due to verifications they do. For example, ES5 15.2.3.5-4-214.js:

function testcase() {
        var descObj = {
            writable: ""
        };

        var newObj = Object.create({}, {
            prop: descObj
        });
        var hasProperty = newObj.hasOwnProperty("prop") && typeof newObj.prop === "undefined";

        newObj.prop = 121;

        return hasProperty && typeof newObj.prop === "undefined";
    }
runTestCase(testcase);

Similar cases exist when ensuring a non-extensible object can't be extended and a non-configurable property can't be deleted.

I don't have an exact count of cases affected but I can confidently say it's over 100.

A quick fix might be to tag these tests noStrict, but ideally the verification steps should be fixed to work in strict and non-strict.

from test262.

I think at this point we can track individual test failures as bugs. Thanks to lots of work from @anba and @smikes we should be mostly strict clean right now!

from test262.

we should be mostly strict clean right now!

I don't want to be the bearer of bad news, but there are still more than 1200 tests failing when running in strict mode. 😢

from test262.

@anba noooo! I guess I messed up my harness command or something :(

from test262.

Just FYI
Test failures: https://gist.github.com/anba/98fff692ea5b2e7a6358
With stacktraces: https://gist.github.com/anba/c46ce3357cbab4b92be6

from test262.

@anba I see those too. I believe there is a race in test262-harness that causes failures when many tests are queued, because when I rerun them individually they succeed, eg

$ test262-harness test/built-ins/RegExp/*.js
[lots of failures]
FAIL test/built-ins/RegExp/S15.10.3.1_A2_T2.js
     Checking if using dafined variable "x = 1" as flags leads to throwing the correct exception
     Exp: no error
     Got: ReferenceError: __patterns is not defined
       at evalmachine.<anonymous>:13:69
       at Socket.<anonymous> (/Users/smikes/src/github/test262-harness/lib/runners/nodehost.js:33:12)
       at Socket.emit (events.js:95:17)
       at Socket.<anonymous> (_stream_readable.js:764:14)
       at Socket.emit (events.js:92:17)
       at emitReadable_ (_stream_readable.js:426:10)
       at emitReadable (_stream_readable.js:422:5)
       at readableAddChunk (_stream_readable.js:165:9)
       at Socket.Readable.push (_stream_readable.js:127:10)
       at Pipe.onread (net.js:528:21)

Ran 407 tests
373 passed
34 failed
Took 2.371 seconds
$ test262-harness test/built-ins/RegExp/S15.10.3.1_A2_T2.js 
Ran 1 tests
1 passed
0 failed
Took 0.102 seconds

from test262.

@smikes __patterns only appears in RegExp/15.10.2_A1_T1, so the file name in the harness output is definitely wrong. Also: Running 15.10.2_A1_T1 with the --testStrict option should report a failure for strict mode. That works for me when using -r jsshell. It did not work for me with -r node and -r node-ip.

Update: It's still broken with [email protected], but it works as expected in [email protected]

from test262.

What's the status of this?

from test262.

What's the status of this?

The numbers from my earlier comment still apply, so ~1200 failing tests.

from test262.

@anba, any comments on how much further to go after your latest PRs?

from test262.

@anba, any comments on how much further to go after your latest PRs?

• ~240 files in built-ins/Object: All remaining Object tests require changes to use propertyHelper functions.
• 72 files in language: I still need to decide how to handle tests using eval. I think I'll convert eval("var x = 0;"); if (x !== 0) { ... } to var result = eval("var x = 0; x); if (result !== 0) { ... }, but I haven't yet made a final decision. (See language/line-terminators and language/white-space for examples where this is necessary.)

from test262.

I can take on the Object tests , if that would help.

from test262.

I can take on the Object tests , if that would help.

Sure. Only a handful of test cases were changed in #249, the rest still needs some work.

from test262.

Working on that this morning.

from test262.

What's left? :-D

from test262.

What's left? :-D

All tests work now in both modes! 😄

The only thing left is updating the python runner to use the new default.

diff --git a/tools/packaging/test262.py b/tools/packaging/test262.py
index b9172b0..ed8939e 100755
--- a/tools/packaging/test262.py
+++ b/tools/packaging/test262.py
@@ -66,9 +66,7 @@ def BuildOptions():
                     help="Test only strict mode")
   result.add_option("--non_strict_only", default=False, action="store_true",
                     help="Test only non-strict mode")
-  # TODO: Once enough tests are made strict compat, change the default
-  # to "both"
-  result.add_option("--unmarked_default", default="non_strict",
+  result.add_option("--unmarked_default", default="both",
                     help="default mode for tests of unspecified strictness")
   result.add_option("--logname", help="Filename to save stdout to")
   result.add_option("--junitname", help="Filename to save test results in JUnit XML format")

from test262.

🎉

from test262.

Done!!! 🍰 🍨 🎈 🎉

from test262.

@anba, @smikes, thanks! ❤️

from test262.