tc39 / test262 Goto Github PK

These are some errors I see with the current test suite in IE11; just aiming to document them for now. They seem to mainly related to the operation of the delete keyword/operator.

S8.7.1_A2 Try to delete y, where y is var y=1 Fail

S8.7_A5_T1 Delete referenced object, var __ref = obj Fail

S10.4.1_A1_T1 Checking if deleting variable x, that is defined as var x = 1, fails Fail

S11.4.1_A3.1 Checking declared variable Fail

S11.4.1_A3.3 Checking declared variable Fail

S12.2_A2 Checking if deleting global variables that have the attributes {DontDelete} fails Fail

12.6.4-2 The for-in Statement - the values of [[Enumerable]] attributes are not considered when determining if a property of a prototype object is shadowed by a previous object on the prototype chain Fail

S13_A12_T1 Checking if deleting a function that is declared in global scope fails Fail

15.3.5.4_2-13gs Strict mode - checking access to non-strict function caller from strict function (indirect eval used within strict mode) Fail

15.3.5.4_2-7gs Strict mode - checking access to non-strict function caller from strict function (Function constructor defined within strict mode) Fail

15.3.5.4_2-9gs Strict mode - checking access to non-strict function caller from strict function (New'ed Function constructor defined within strict mode) Fail

15.5.4.9_CE Tests that String.prototype.localeCompare returns 0 when comparing Strings that are considered canonically equivalent by the Unicode standard. Fail

S15.5.5.1_A3 Checking if deleting the length property of String fails

The following files contain a 'path' entry in the 'flags' sequence. Is that just a leftover from the conversion process?

• es6/Array.prototype.find/Array.prototype.find_callable-predicate
• es6/Array.prototype.find/Array.prototype.find_empty-array-undefined
• es6/Array.prototype.find/Array.prototype.find_length-property
• es6/Array.prototype.find/Array.prototype.find_modify-after-start
• es6/Array.prototype.find/Array.prototype.find_non-returning-predicate
• es6/Array.prototype.find/Array.prototype.find_noncallable-predicate
• es6/Array.prototype.find/Array.prototype.find_predicate-arguments
• es6/Array.prototype.find/Array.prototype.find_push-after-start
• es6/Array.prototype.find/Array.prototype.find_remove-after-start
• es6/Array.prototype.find/Array.prototype.find_return-found-value
• es6/Array.prototype.find/Array.prototype.find_skip-empty
• es6/Array.prototype.find/Array.prototype.find_this-defined
• es6/Array.prototype.find/Array.prototype.find_this-is-object
• es6/Array.prototype.find/Array.prototype.find_this-undefined

Basically there a bunch of tests that assign to indexes on the global object. This is something that is invalid. https://bugs.ecmascript.org/show_bug.cgi?id=1453
It would be awesome if somebody could remove or modify these tests to not run in a browser environment.

It would be nice to map a feature name to a set of tests. We could see how many tests exist for each feature and get a breakdown of test pass rates. Would also be nice to pass a feature name to the harness rather than a glob (eg. > test262 Array#contains --prelude containsShim.js)

Is the file system the best way to do this or should we consider an additional front matter attribute?

The HasProperty check was removed in rev27.

Two tests need to be updated.

• es6/Array.prototype.find/Array.prototype.find_remove-after-start
• es6/Array.prototype.find/Array.prototype.find_skip-empty

I have started work on translating the Promises/A+ tests into a format that works with Test262.

I have a branch for this work -- https://github.com/smikes/test262/tree/promises-aplus-tests-1/test/suite/promises-aplus -- where I created a harness file, a subdirectory test/suite/promises-aplus for the tests, and a README file documenting the approach that I'm taking.

The first thing I am working out is how to put operations into a particular order without using setTimeout; the Promises/A+ tests use this at times, e.g., to guarantee that a particular call to resolve or reject will happen after all currently queued Promise jobs.

I would be very grateful for feedback about my approach.

/cc @domenic @getify @anba and @niksurya

Apologies if I've missed it or I'm off track, but I was looking for a test tracking the feature that let declarations in a for (;;) style loop are fresh bound per iteration, and I wasn't able to find it. For that matter, I'd also like to see the test for fresh binding for for-in loops too.

If these tests exist and I missed them, I apologize. I spent about 30 minutes combing through and it became painfully obvious I didn't know what exactly to be looking for.

This code from CONTRIBUTING.md

var error;

try {
    [test code]
} catch (e) {
    error = e;
}

if (error === undefined) {
    $ERROR('expected to throw an error but no error was thrown');
} else if(!(e instanceof ReferenceError)) {
    $ERROR('expected to have ReferenceError, got ' + error.name + ' instead.');
}

is frustrating to repeat. Can we turn it into

assertThrows(ReferenceError, function () {
    [test code]
});

via a helper?

The test262.py script does not list the 'Expected to fail but passed' list, when --logname is set. It failes with the following error.

Expected to fail but passed ---
Traceback (most recent call last):
File "./tools/packaging/test262.py", line 629, in
code = Main()
File "./tools/packaging/test262.py", line 624, in Main
options.junitname)
File "./tools/packaging/test262.py", line 553, in Run
self.PrintSummary(progress, logname)
File "./tools/packaging/test262.py", line 508, in PrintSummary
self.logfile.append(" %s in %s \n" % (result.case.GetName(), result.case.GetMode()))
AttributeError: 'TestSuite' object has no attribute 'logfile'

Running on windows 7 with V8 version 3.28.32 (candidate) [sample shell]

Z:\Code\github\test262>tools\packaging\test262.py --unmarked_default=both --comm
and="v8" 15.5.4.9_CE
=== ch15\15.5\15.5.4\15.5.4.9\15.5.4.9_CE failed in strict mode ===
=== ch15\15.5\15.5.4\15.5.4.9\15.5.4.9_CE failed in non-strict mode ===

Both cause "This application has stopped working" errors; in debugger I get

Unhandled exception at 0x014014e6 in v8.exe: 0xC0000005: Access violation reading location 0x00000000.

Access violation reading NULL pointer at i18n.cc line 537.

I will update my v8 and retest.

Expected test results are wrong in:

• es6/ch22/22.1/22.1.3/S22.1.3.6_T1.js

Suggested fix:

--- a/test/suite/es6/ch22/22.1/22.1.3/S22.1.3.6_T1.js
+++ b/test/suite/es6/ch22/22.1/22.1.3/S22.1.3.6_T1.js
@@ -10,7 +10,7 @@ runTestCase(function () {
   var testArr = new Array('testString', 'anotherTestString', 3),
       updatedArr = testArr.fill('newValue', 1, 3);

-  if (updatedArr[3] !== 'newValue') {
+  if (updatedArr[3] !== void 0) {
     return false;
   }

@@ -18,11 +18,15 @@ runTestCase(function () {
     return false;
   }

+  if (updatedArr[1] !== 'newValue') {
+    return false;
+  }
+
   if (updatedArr[0] !== 'testString') {
     return false;
   }

-  if (updatedArr.length !== 4) {
+  if (updatedArr.length !== 3) {
     return false;
   }

Support request for @smikes, basically.

I added my Array.prototype.contains tests to a local V8 checkout, into which I compiled an implementation. All of the positive tests pass, but all of the negative tests fail.

The command I ran was

$ ./tools/packaging/test262.py --command "/c/src/v8/build/Debug/d8 --harmony" Array.prototype.contains

The failing results look like:

=== es6\Array.prototype.contains\Array.prototype.contains_exception-before-match was expected to fail in non-strict mode
, but didn't ===
--- expected error: Test262Error ---

--- output ---
 c:\users\domenic\appdata\local\temp\test262-knixpu.js:62: Test262Error: This error should be re-thrown
        throw new Test262Error('This error should be re-thrown');
              ^===

This kind of indicates the way that it's detecting Test262 errors is off, because they are being thrown, but not detected.

I know there was a lot of work on this recently; is there a chance it broke?

Don't know if this the right place to do this or not, but here goes:

proposal:

since we are betraying the ES grammar/syntax in a mayor way. I will like to introduce a new... more identical familiar way of writing a function:

function()

and that's it.

the ideal came to me from the spite I have from the change of the spec grammar.

Example:

var bar;
bar = [];
[0,1,2,3,4,5,6,7,8,9].forEach(function(arg)bar.push(x));

function foo(i) console.log(i);

Surprising.... this work... in Firefox. But only in firefox it work... not fully sure, but I know it does not work in google chrome.

Will that's it!

Comment, email, or close, you have control.

a note to add this is the same rules following single statment whre:

for()
if()

follow.. if it's on a single line, than the body --open and close curly brace-- can be omitted.

There are a number of tests for typed arrays in the Khronos suite. Some of them are out of date (eg. requiring that DataView called as a function should throw), but most should be applicable to the current ES6 draft.

Typed array tests: https://github.com/KhronosGroup/WebGL/blob/master/sdk/tests/conformance/typedarrays

As usual, if porting is going to be problematic for whatever reason we can get coverage from scratch, but a lot of the Khronos tests are fairly in-depth so I'd like to use them if at all possible.

https://github.com/tc39/test262/blob/master/test/suite/es6/Array.prototype.find/Array.prototype.find_this-is-object.js

In strict mode, all of these should be passed through verbatim, not boxed.

The chapter numbers do not correspond between 5.1 and 6.0; for example, in ES5.1 section 7.2 is about lexical whitespace, while in 6.0 section 7.2 is for operators.

Where should ES6 tests go? Probably in their own hierarchy, (as opposed to a branch) so that selected ES5 tests can be applied to an ES6 implementation.

V8 has support for this (for now): https://code.google.com/p/v8/source/detail?r=24783 --- The test cases I wrote for this might be suitable for adaptation into the ES6 test suite.

There might be a few missing, so it would be good to get a look at it.

Possibly in the form of a grunt build system task? I'm suggesting this because policing consistent style is not going to scale across all of the files here. We should just agree on some defaults and let the machine handle the rest.

The following files don't test for the correct value.

• es6/Number.isNaN/Number.isNaN_Boolean.js
• es6/String.prototype.endsWith/String.prototype.endsWith_Fail.js
• es6/String.prototype.endsWith/String.prototype.endsWith_Fail_2.js

Currently the test code is scanned for $INCLUDE() directives. Inclusion should be marked in the frontmatter instead.

What syntax should be used?

We are seeing inconsistent behavior between major engines over in TreehouseJS/jsdom@15e3c4f

In ES6, array methods can operate on lengths up to 2^53 - 1, instead of 2^32 - 1. This makes writing tests hard. Consider:

if (Array.prototype.contains.call({ length: +Infinity }, 'a') !== false) {
    $ERROR('Expected { length: +Infinity } to not contain \'a\'');
}

var arrayLikeWithTrap = {
    length: +Infinity,
    get 9007199254740992() {
        $ERROR('Getter for 9007199254740992 (i.e. 2^53) was called');
    },
    '9007199254740993': 'a'
};

if (Array.prototype.contains.call(arrayLikeWithTrap, 'a') !== false) {
    $ERROR('Expected trapped array-like with length 9007199254740992 to not contain \'a\'');
}

var arrayLikeWithTooBigLength = {
    length: 9007199254740995,
    '9007199254740992': 'a'
};

if (Array.prototype.contains.call(arrayLikeWithTooBigLength, 'a') !== false) {
    $ERROR('Expected array-like with too-big length to not contain \'a\', since it is beyond the max length');
}

all of these tests will likely take longer than the age of the universe to complete, since they are testing an algorithm that takes 2^53 iterations to reach the case we care about.

I'd be curious about other strategies I should employ instead?

The following tests assume NaN === NaN is true.

• es6/Math.fround/Math.fround_NaN.js
• es6/Math.hypot/Math.hypot_NaN.js
• es6/Math.trunc/Math.trunc_NaN.js

ES5 has the following to say:

The characters / or backslash \ occurring in the pattern shall be escaped in S as necessary to ensure that the String value formed by concatenating the Strings "/", S, "/", and F can be parsed (in an appropriate lexical context) as a RegularExpressionLiteral that behaves identically to the constructed regular expression. For example, if P is "/", then S could be "/" or "\u002F", among other possibilities, but not "/", because /// followed by F would be parsed as a SingleLineComment rather than a RegularExpressionLiteral. If P is the empty String, this specification can be met by letting S be "(?:)"

The key is that var r = new RegExp(str); eval("/" + r.source + "/") should give you a regexp identical to r.

Implementations don't follow this too closely. IE doesn't handle empty string properly. FF/Chrome don't seem to handle "\n" properly. There are probably other interesting cases to be discovered.

Consider the case where Promise.all([p1, p2]) is called, and p1 and p2 are both unsettled (non-foreign) promises. Then Promise.all installs an onFulfilled handler on p1, p2 which is a Promise.all Resolve Element Function

Assuming both p1, p2 eventually are fulfilled, whichever one fulfills second triggers the final steps of that algorithm, specifically:

 10 If remainingElementsCount.[[value]] is 0,
  a Let valuesArray be CreateArrayFromList(values).
  b Return the result of calling the [[Call]] internal method of promiseCapability.[[Resolve]] with undefined as thisArgument and (valuesArray) as argumentsList.
 11 Return undefined.

CreateArrayFromList creates an Array object. If Array.prototype.then exists and is callable, then instead of fulfilling the Promise with the array, the "foreign thenable" code path in promiseCapability.[[Resolve]] will be exercised, and the Promise returned from Promise.all will be "locked in" to whatever thenable Array.prototype.then looks like.

It's hard for me to think of a good design reason that I would want to add a method then to Array.prototype or Object.prototype, so this is not something most people will encounter in normal development.

Now this is all obviously insane, and possibly just a minor nuisance, but there may be a security risk here in that an attacker who can modify Array.prototype (which is already quite bad, I understand!) would be able to break any Promise.all resolutions in the same ... Realm? Vat? execution context, maybe.

/cc @domenic
Edited above to clarify meaning.

While many tests are failing spuriously in light of ES6 changes, we may take some time to update them - current consensus was to update once the spec is ratified, and while I think @anba has captured many of the changes in pull/29, I have yet to review it closely 😦. However I think we should remove now the tests that freeze new implementations so at least people can still run the official suite (even though the numbers are less meaningful). The only two I am aware of are 15.4.4.5_A4_T1 and 15.4.4.5_A4_T2 (due to toLength change these are no longer boundary value tests). I plan to remove these from the es5 branch next week and update the public website (once I figure out how).

Test262 has many tests that test the interaction between two features. For example, proxies are generally interesting values and so any API should be tested with proxy values. This has the unfortunate effect of failing those API tests when proxies aren't available, which doesn't make much sense.

One way to solve this problem is specifying a prerequisite.

Two ways I can think of handling this:

1. specify a function (somehow) that must return true before a test is run

2. specify prerequisites in the front-matter

   a. map feature names to existence checks for said feature
   b. map feature names to the collateral for that feature, and only run the test if the tests have a pass rate higher than a certain threshold.

1 is hard to support because we need some way to pass a function from the test to the harness, which isn't something we have today and seems difficult (especially when running tests out-of-proc where you can't just pass the harness a reference to the prereq function).

2.a shouldn't be too hard to support, but since we want to support mapping feature names to test collateral for that feature (#107), 2.b seems somewhat interesting to pursue.

Thoughts?

When I reviewed the async test code I believe I saw a provision for overriding the default timeout value by setting the timeout attribute on a test -- either via @timeout 100 (old-style) or timeout: 100 (new-style).

Someone (me?) should verify that this exists and document it so that it can be used.

Also, some tests may be designed to time out (e.g., a test of a never-fulfilled promise), so there should be a documented way of writing an async test which is expected to time out.

All of this is necessary because we must avoid the use of setTimeout in Test262 tests.

The license/copyright header needs to use single-line comments, otherwise tools/packaging/common.py and tools/packaging/parseTestRecord.py won't work.

Can feature detect browsers by checking for Annex B FIB behavior.

Number.toInteger() was removed from the ES6 draft specification, this test file should be removed, too.

• es6/Number.toInteger/Number.toInteger_Double.js

Following on from: (Clean-up 'simple test harness')[https://bugs.ecmascript.org/show_bug.cgi?id=41]

The simple test harness code in sth.js and helper.js is reasonably clean but lacks unit tests and mixes logic with presentation.

I propose to split the code into five separate files -- one each for the four main classes (BrowserRunner, TestLoader, Controller, Presenter) and one for the main function that sets everything up (anonymous function on sth.js line 682). I would then add unit tests for each of the classes, refactoring when possible.

I would modify packager.py so that it concatenates these files together into a single file (e.g., test-harness.js) and make the generated website .html files depend on this new file.

Ultimately a modern build system might be used to concatenate, minify, gzip, etc. etc. all the frontend files. But for the moment, just refactoring and adding tests for existing behavior is enough.

I will start work on this after the #51 PR is disposed of.

Currently tests use a custom syntax for adding attributes to tests. Unfortunately it is difficult to parse and generate and there are no libraries to make this easier. I have a number of attribute normalizations I want to do to make harnessing easier and simplify test authorship, so now might be a good time to change things.

I propose we convert custom format to a YAML Associated List of key-value pairs, with all "flags" moved to a flags array. An example:

/**
 * The production Block { } in strict code can't contain function
 * declaration;
 *
 * @path bestPractice/Sbp_A1_T1.js
 * @description Trying to declare function at the Block statement
 * @onlyStrict
 * @negative SyntaxError
 * @bestPractice
 * http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls
 */

gets converted to:

/*---
info: >
    The production Block { } in strict code can't contain function
    declaration;
description: Trying to declare function at the Block statement
negative: SyntaxError
bestPractice: "http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls"
flags: [onlyStrict]
---*/

JSON has also been suggested over YAML, which I am amenable to except for the excess punctuation and lack of multi-line strings. Some info records are >500 characters so the line length would get absurd.

/*{
"info": "The production Block { } in strict code can't contain function declaration;",
"description": : "Trying to declare function at the Block statement",
"negative": "SyntaxError",
"bestPractice": "http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls",
"flags": ["onlyStrict"]
}*/

If we can approve this I can move rapidly on the normalizations, including making sure all negative attributes specify the kind of error the expect, moving $INCLUDE's to an includes collection in the front-matter, adding es5id and es6id attributes to track spec sections in our various spec editions (currently only reflected by file path), and etc.

Thoughts?

The following frontmatter parses cleanly:

description: ensure correct value for 'this'

The following causes a test error:

description: 'this' must have correct value

I was surprised by this, and would like to make a note of it here. I believe this is a YAML feature. To avoid problems, don't start scalar fields with a single-quote.

We need a higher-level assert library than just using ifs and $ERROR. This will help those coming from other test frameworks, whether they be shim authors or implementers.

Using Node's assert as a base, here's what's good/bad there:

• "equal" meaning == is horrible. In fact we should just use more precise terms like abstractEqual, strictEqual, sameValueZero, sameValue.
• deepEqual is too high-level and complex for us; semantics are unclear. We might want iteratorEqual or something but probably not, or at least not yet.
• the overloads of throws are a bit annoying. And in general we prefer negative tests for those. Maybe leave it out initially? If we do include it, I have some ideas for how to make it better.
• ifError is not that useful for tests; it's more for callback-based code.
• fail is redundant with our $ERROR.

So, initial set:

• assert(truthy, message)
• assert.sameValue(actual, expected[, message])
• assert.notSameValue(actual, expected[, message])

Candidates to consider on an as-needed basis:

• assert.throwsWithName(callback, exceptionName[, message])---literally checks the .name property of the thrown exception object
• assert.throwsExpected(callback, expectedException[, message])---checks thrown exception is same-value expectedException
• assert.arrayLikeEquals(expected, actual[, message])---checks .length properties and if they match does a for loop to check samevalue for each element. This is 80% of my use cases for deep equal.

What else?

Number.prototype.clz was replaced with Math.clz32.

CC @anba

I believe we can make this work just by removing the line which sets negative to "." Will try this out later.

Some test descriptor entries only appear in a limited amount of test files, maybe these should be removed resp. normalised.

• assertion, section, name
  • Only used in S15.2.3.1_A3
• email
  • Only used in tests contributed by ryanhlewis
• spec
  • Only used in new es6 tests, contains links to http://people.mozilla.org/~jorendorff/es6-draft.html

Can you think of a simple way to support your assert while also showing useful debug information for console hosts, eg. error thrown and line number? I suppose some hosts could define assert such that it throws, and others could report the error and allow the test to continue... But not sure we want hosts to differ in how they execute tests...

Continuing the digression from #22

Yes, I think I can. Rather than using a JSON protocol to communicate about test success, we could use TAP. A test still ends when an exception is thrown and uncaught, leaving error and line number and stack trace information intact for those who can see it. But within the test, the assertions can generate TAP output.

A console runner would write strings to stdout. A browser runner would accumulate strings in an array.

The test harness would need to be augmented a bit to preload the expected number of assertions, but this could be precalculated by the packager and stored in the metadata (or frontmatter, but we don't want to make test authors maintain it.)

So a simple test would produce short TAP output:

1..1
ok 1 - threw ReferenceError on "1 += 1"

A complex test would produce more output:

1..48
ok 1 - message 1
...
ok 48 - message 48

And a failing test, in addition to a throw, would produce truncated TAP output:

1..20
ok 1 - message
not ok 2 - failure message

where the facts that a) an assertion failed and b) 18/20 assertions were not tried would provide finer information about standard conformance than we currently capture.

However this project is bigger than anything I want to start at the moment, so I will leave it at this. Just wanted to document the thought.

If a test doesn't indicate that it's specifically for sloppy or strict mode it should be run in both. This change should be made at least for the web harness, but also perhaps for the python harness.

In order to test module loaders and promises, test262 needs to support asynchronous tests.

The most straightforward way would be to extend the signature of runTestCase(fn) The test case function could get a done function as a parameter, similar to what Mocha does.

I think this would introduce the least increase to the API surface.

An open issue is what to do with tests that time out, because test262 should not know about setTimeout.

@domenic @kriskowal would you mind commenting on this?

Multi-line comment not properly terminated in es6/ch22/22.1/22.1.2/S22.1.2.1_T3.js.

intl402/ch11/11.1/11.1.1_20_c.js needs to be updated to change the expected minor unit value for CLF (Unidad de Fomento) from 0 to 4.

The URL in the test file should also be updated, from "http://www.currency-iso.org/dl_iso_table_a1.xml" to new the location at "http://www.currency-iso.org/dam/downloads/table_a1.xml".

See: http://www.currency-iso.org/dam/isocy/downloads/dl_currency_iso_amendment_157.pdf
See Also: https://bugzilla.mozilla.org/show_bug.cgi?id=946528

This list is from https://bugs.ecmascript.org/show_bug.cgi?id=1453

/cc @evilpie

Current Status is with SpiderMonkey Version: JavaScript-C34.0a1 compiled by me on Windows 7.

All tests pass in SpiderMonkey. Will set up my server and get test result from Firefox.
... and all tests still fail in Firefox.

Apologies if this isn't the correct place for reporting.

It would be really helpful if the test262 website or wiki mentioned the new Github location of the test262 code. The old Mercurial repository seems to be defunct. Here are a couple of places that mention the old location:

http://wiki.ecmascript.org/doku.php?id=test262:command
http://wiki.ecmascript.org/doku.php?id=test262:submission_process

The only place that I could find that mentioned the new location was in a mailing list message, and this message was pretty hard for me to find.

It could, e.g., contain a brief FAQ: What is test262? How to contribute? Etc. The content on this page would be perfect; I think it would be worth the redundancy.

Ref case #74

I would like to do this in one of my tests:

Number.prototype[0] = "a";
Number.prototype[1] = "b";
Number.prototype.length = 2;

Do I have to clean up afterward, or can I assume that each test file is independent?

Documenting this in CONTRIBUTING.md is what I am really after :)

A number of tests were rewritten to use negative: Test262Error instead of throwing NotEarlyError (c33bf0e). I don't quite understand why this change actually works. Applying ToString() on Test262Error objects returns a string with the prefix "Test262 Error:" (see Test262Error.prototype.toString). And applying the requested error pattern /Test262Error/ against "Test262 Error:" should always fail (cf. error detection in $DONE).

Am I missing something here?

@mathiasbynens has some rad Annex B string tests over on mathias.html5.org.
It would be great to port them over to test262.