inexorabletash / text-encoding Goto Github PK
View Code? Open in Web Editor NEWPolyfill for the Encoding Living Standard's API
License: Other
Polyfill for the Encoding Living Standard's API
License: Other
to make it easier to include
The version of text-encoding on NPM is 0.5.2 and quite old. The current version contains some very big performance improvements. I would be great if someone could put the current version on on NPM.
The current unit tests pass in all parameters, but several of the parameters are specified (and implemented in the shim) as optional. Verify that the optional parameter behavior is per spec.
I'd like to use the new NONSTANDARD_allowLegacyEncoding
option, but it hasn't been released in a version on npm yet.
Hi, and first of all thanks for this wonderful library. Much needed to export CSV files in CP1252 in my case. Fun!
I'm using your library like this : var array = TextEncoder('windows-1252').encode(text);
To make it work, I had to change encoding.js:933
to suppress the checks forcing the encoding name to be either 'utf-8', 'utf-16le' or 'utf-16be'. I'm not sure why this is needed, but I'm probably terribly wrong.
Anyway, removing this makes it work perfectly. Will generate a pull request so you can read the diff more easily. Thanks again!
Decoders for these are implemented but untested:
I'm working on a tool to generate a CSV export of some data in a browser, and unfortunately, the encoding situation is a bit of a mess. TL/DR, it seems that the best outcome results in a tab separated, utf-16-le encoded file. [1][2]
I'm interested in using this package not as a polyfill, but by directly loading it when I need to perform that encoding. Would you be open to a PR that moved the core encoding functionality into a standalone module, and implementing the polyfill as a consumer of that standalone library?
[1] https://donatstudios.com/CSV-An-Encoding-Nightmare
[2] http://terminaln00b.ghost.io/excel-just-save-it-as-a-csv-noooooooo/
new TextEncoder().encode(false) // Chrome, Firefox: [102, 97, 108, 115, 101]
new TextEncoder().encode(0) // Chrome, Firefox: [48]
new TextEncoder().encode(false) // Polyfill: []
new TextEncoder().encode(0) // Polyfill: []
It would be nice to be able to only have the Encode or Decode code, the code is really long and for my actual project that must be as small as possible I am needing only the Encode part, is this possible to do?
I have Installed this package over Bower
bower install text-encoding
It is in my index.html
...
<body ng-app="myApp">
<script>
window.TextEncoder = window.TextDecoder = null;
</script>
...
<!-- build:js({client,node_modules}) app/vendor.js -->
<!-- bower:js -->
<script src="bower_components/text-encoding/lib/encoding.js"></script>
<script src="bower_components/text-encoding/lib/encoding-indexes.js"></script>
<!-- endbower -->
Iam using it here
case 'php':
...
var uint8array = new TextEncoder(
'windows-1252', { NONSTANDARD_allowLegacyEncoding: true }).encode(str);
data = new Blob([uint8array], {
type: 'application/json; charset=windows-1250;'
});
this.FileSaver.saveAs(data, this.$scope.selectedLang[i].code + '.php');
}
It throws the Error
Encoder not present. Did you forget to include encoding-indexes.js?
Where did I something wrong?
Removing
<script>
window.TextEncoder = window.TextDecoder = null;
</script>
Errors then
Failed to construct 'TextEncoder': The encoding provided ('windows-1252') is not one of 'utf-8', 'utf-16', or 'utf-16be'
So it finds it but cant use because of the Pollyfill ?
So did I placed the
<script>
window.TextEncoder = window.TextDecoder = null;
</script>
Somewhere wrong? Where have it to be?
Since the upgrade it gets stuck in an endless loop.
I know the license is the BSD-3-clause one, but what worth is a license mentioning the copyright holders if they're not made explicit?
I can't help but notice that testharness.js looks like a mix of the W3C version and it's api -- so perhaps a copyright to the W3C is correct?
I am not able to reproduce this error myself, but I have received several issue reports to my shapefile parser of an error generated by text-encoding: 1 2 3. The error is:
Decoder not present. Did you forget to include encoding-indexes.js first?
But since the Node API is simply require("text-encoding")
, the error is internal, because the text-encoding library should be loading the indexes itself in Node.
In an attempt to workaround this error, I reverted shapefile to use [email protected] rather than 0.6.3, as it appears that changes 8c43765 b8503e7 may be related. Sorry I don’t have any more information to go on. If users report any more details I will relay them here. Thank you!
Just thinking it would be nice to have a CP437 (IBM US-DOS Extended ASCII) convertors available.
There have been a number of projects to convert ansi/ascii art for display in modern UTF systems... It would be a nice to have feature if this were available here... I'm not sure about the format used in the lib/encoding-indexes.js
file... Is the array simply a mapping of position X to character code/pair Y? Does this start at the byte value of 1 (skipping null)? And am I correct in assuming I can go all the way from character 1 through 255, and does null mean don't map, or default map?
I'd be happy to make a PR if these questions could be clarified... possibly adding comments as to the structure of the encoding-indexes
in the file itself.
I couldn't find a license in the repo. What is the license for this project? BSD, MIT?
Thanks
I am just curious, what was the reason to make 630eed1 happen?
I was about to request changing to MIT license from Unlicense, but instead opened this question...
A lot of tool chains are moving towards using ES6 modules (rollup for example) and having ES6 modules published to npm would make this package easier to consume.
Getting Uncaught TypeError: Cannot read property 'ibm866' of undefined
error in client-console. It's likely an issues occurs due to expectation that encoding-indexes.js
returns an object with encoding-indexes whereas it assigns them to global object.
Hi,
We depended on ^0.5.2, and our CI server just installed 0.5.5, which broke the build. We received an exception thrown from here https://github.com/inexorabletash/text-encoding/blob/master/lib/encoding.js#L979.
The solution is simple enough (a change from TextDecoder('utf-8').decode(...)
to creating an instance of var decoder = new TextDecoder('utf-8'); decoder.decode(...)
), but it caused us some debug time to fix because the documentation isn't up to date, and the breaking change was introduced in a patch release.
Was this intentional, or are the documentation updates just pending?
Thanks!
Track the spec update: whatwg/encoding@e32a57b
Bug: whatwg/encoding#73
The current tests don't validate the behavior of the implementation when invalid parameters are passed. Add tests for these cases.
Using https://nodejs.org/api/string_decoder.html
should theoretically significantly decrease size of manually supported code as it already provides proper decoding algorithms for popular (supported in Node) encodings.
I am having problems encoding Shift_JIS text, If I try to encode a character (in my case "点"), instead of getting 2 bytes I get 3 incorrect bytes. Decoding seems to work ok. Am I missing something?
Test code:
const textEncoder = new TextEncoder("Shift_JIS")
const textDecoder = new TextDecoder("Shift_JIS")
const text = "点"
console.log("Decode from bytes:")
const bytes = Uint8Array.from([0x93,0x5f])
const textResultFromBytes = textDecoder.decode(bytes)
console.log(bytes, "->", textResultFromBytes, textResultFromBytes === text ? "OK": "NOT OK")
console.log("Encode text then decode again:")
const bytesOfText = textEncoder.encode(text)
const textResultFromText = textDecoder.decode(bytesOfText)
console.log(text, "->", bytesOfText, "->", textResultFromText, textResultFromText === text ? "OK": "NOT OK")
Console output:
Decode from bytes:
Uint8Array(2) [147, 95] "->" "点" "OK"
Encode text then decode again:
点 -> Uint8Array(3) [231, 130, 185] -> 轤ケ NOT OK
This quick test is also available here:
https://jsfiddle.net/aleris/pt17dkz3/
I found a problem when trying to encode a string with turkish characters in windows 1254:
TypeError: Cannot read property 'indexOf' of undefined
at indexPointerFor (encoding.js:823)
at SingleByteEncoder.handler (encoding.js:1434)
at Object.encode (encoding.js:1122)
Way to reproduce:
var textEncoder = new CustomTextEncoder('windows-1254', {NONSTANDARD_allowLegacyEncoding: true}),
contentEncoded = textEncoder.encode(["Tuğce"]);
I am using stringencoding in one of my projects... but it feels pretty dead to me
https://www.npmjs.com/package/stringencoding
https://github.com/defunctzombie/stringencoding
Is seems text-encoding is an updated fork, can someone explain the relation between stringencoding and text-encoding, what is the functional difference.
Hi!, I'm a Chinese.
when I use the library like :
var dataString = '"操作","运单状态","运单号","货物名称","件数","寄件客户","寄件人电话","寄件地址","收件客户","收件人电话","收件地址","寄件网点","目的地","目的网点","重量(kg)","体积(M³)","到货时间","包装","到付款","代收货款","回付金额","寄货日期","回单号","回单","回单备注","提货密码"'
var textEncoder = new CustomTextEncoder('gbk', {NONSTANDARD_allowLegacyEncoding: true});
var csvContentEncoded = textEncoder.encode([dataString]);
var blob = new Blob([csvContentEncoded], {type: 'text/csv;charset=gbk;'});
Uncaught TypeError: The code point 179 could not be encoded.
some words could not be encoded ?
I found that 'M³'
in the words caused, 'M'
is ok, can support 'M³'
?
... which really just means: sync to the indexes.json file
Could you please release the 0.6.2 version to npm?
Whenever I try to use encoding.js
without encoding-indexes.js
it throws this error. It didn't used to be like that, it was optional, if I recall correctly. Did something change?
If it is required, why not just put the files together?
...not just ArrayBufferView, per spec (via WebIDL/BufferSource)
I'd like to get these APIs implemented in Servo, but I'm unwilling to do one-off test imports. @inexorabletash, are you willing to do the work on this?
The npm package is still at version 0.6.2. Could you publish it?
Hi all,
chrome has its own implementation of TextEncoder(), but it doesn't support the legacy encoding.
I use your implementation simply replacing "TextEncoder" with "NewTextEncoder" directly editing your source code.
There is a better way?
Thanks
I'm serializing stuff to a C++ library that wants null terminated strings. I noticed a "TODO: any options?" in your code, and this seems like a candidate. Would a pull request offering such an option be welcomed?
Add MS932 as alias for shift-jis:
whatwg/encoding@01db1f8
Right now encoding-indexes.js
is always required in a node like environment (such as browserify).
This was done in a85ce6f
It would be better if encoding-indexes.js
was never implicitly included.
I would love to see this feature polyfill if (and only if) I need it!
Could you please tag releases, so github makes release tarballs?
Thanks!
Using something like https://ci.testling.com/. That way on every pull request we can get pass/fail information on various browsers.
The option NONSTANDARD_allowLegacyEncoding: true works fine in Chrome and Internet Explorer but is been ignored in Firefox.
The following GB18030 decode incorrect.
=?GB18030?B?lDnaMw==?=:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.