"Replace" vs. WHATWG error handling about rust-encoding HOT 6 CLOSED

Aargh, I should have written the rationale in the README. ;) First, the different behavior is due to the grouping of the sequences (sequences marked * are invalid, so converted to U+FFFD):

what WHATWG expects:        [BF EC] [BF CD] [FF]* [BE D3]
what rust-encoding expects: [BF EC] [BF CD] [FF BE]* [D3]*

Both are acceptable for most usage. The latter behavior was chosen for simplifying the API; if the former behavior were default, the EncoderError would look like this:

pub struct EncoderError<'self> {
    remaining1: ~str, // a portion of the input consumed by the prior `feed` and should be re-fed
    remaining2: &'self str, // a portion of the input given to the current `feed` and not processed
    problem: ~str,
    cause: ~str,
}

For example, the UTF-8 sequence EA B0 7F is invalid, but if the caller gave EA B0 and 7F in the different calls, the second call should return the remaining bytes starting with B0 7F which first byte is not in the current input. This is rather inconvenient, so I chose to eliminate remaining1 and keep only remaining2 (aka remaining in the current API).

I admit the entire idea of the custom error range is hard (we can't have an arbitrarily large error range anyway, maybe that was the rationale of the WHATWG behavior), and if we can have a better design I strongly vote to revert this behavior.

from rust-encoding.

@annevk, fyi

from rust-encoding.

FWIW, browsers do not seem to have interop on error handling: data:text/plain;charset=utf8,%EA%B0%33 gives �3 in Firefox and ��3 in Chrome.

from rust-encoding.

FWIW, utf-8 error handling is aligned with the best practice recommendation of the Unicode standard (as noted in the Encoding Standard).

from rust-encoding.

I’ll just leave this here: http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

from rust-encoding.

I've got a new error handling semantics compatible to the Encoding standard as of a5b0938. It is intertwined with the removal/renewal of TextEncoder/TextDecoder (#4), so it cannot be merged right now, but this issue is now close to be resolved.

from rust-encoding.