andris9 / encoding Goto Github PK

Encode/decode strings with iconv-lite

License: Other

JavaScript 100.00%

encoding's Introduction

Encoding

encoding is a simple wrapper around iconv-lite to convert strings from one encoding to another.

Initially encoding was a wrapper around node-iconv (main) and iconv-lite (fallback) and was used as the encoding layer for Nodemailer/mailparser. Somehow it also ended up as a dependency for a bunch of other project, none of these actually using node-iconv. The loading mechanics caused issues for front-end projects and Nodemailer/malparser had moved on, so node-iconv was removed.

Install

Install through npm

npm install encoding

Usage

Require the module

var encoding = require("encoding");

Convert with encoding.convert()

var resultBuffer = encoding.convert(text, toCharset, fromCharset);

Where

text is either a Buffer or a String to be converted
toCharset is the characterset to convert the string
fromCharset (optional, defaults to UTF-8) is the source charset

Output of the conversion is always a Buffer object.

Example

var result = encoding.convert("ÕÄÖÜ", "Latin_1");
console.log(result); //<Buffer d5 c4 d6 dc>

License

MIT

encoding's People

Contributors

Stargazers

Watchers

encoding's Issues

iconv-loader webpack issue

As you are aware, this works if library found is iconv, but what if the fallback is on iconv-lite ?

'use strict';

var iconv_package;
var Iconv;

try {
    // this is to fool browserify so it doesn't try (in vain) to install iconv.
    iconv_package = 'iconv';
    Iconv = require(iconv_package).Iconv;
} catch (E) {
    // node-iconv not present
}

module.exports = Iconv;

webpack output

WARNING in ./~/encoding/lib/iconv-loader.js
9:12-34 Critical dependency: the request of a dependency is an expression`

I am using webpack more like a bundler to ship single-file functions at AWS lambda for execution(there is no npm available there) and to also bundle dependencies(binary files are bundled too, by a manual script I maintain)

This change removes any issues:

'use strict';
module.exports = require('iconv-lite').Iconv;

But it will probably blow-apart when the dep is iconv and not iconv-lite, I don't know how to solve this only for my project.

What if the wrapper is more like this ?

'use strict';
var Iconv = null;
// this is to fool browserify so it doesn't try (in vain) to install iconv.
try {
  Iconv = require('iconv-lite').Iconv;
} catch (LE) {
  try {
    Iconv = require('iconv').Iconv;
  } catch (E) {
    // node-iconv not present
  }
}
module.exports = Iconv;

Null bytes not trimmed.

Please excuse me if this is the intended behavior but null bytes are not removed from end of the string.

Example code (it is Encoded correctly however the null bytes are there....)

console.log(encoding.convert('\xC7\xAE\xB1\xD2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00','UTF-8', 'EUC-CN').toString().length);

The word is 钱币 in English Coin.

A simple regex replace seems to handle it,However I am not sure if it is the correct solution.

string.replace(/\0\0.*/,'')

Please add copyright statement and license text

Hi!

I am packaging encoding for Debian, as part of an ongoing effort to package pump.io:
https://wiki.debian.org/Javascript/Nodejs/Tasks/Pump.io

In order for it to be accepted into the Debian archive, I need to be able to point to a copyright and license statement, e.g. in a LICENSE file. The MIT license requires that both the copyright statement and the conditions are included when redistributing the source code.

Please could you add this to node-encoding? Committing to git would be sufficient.

Thanks in advance,

Tim

Critical dependency: the request of a dependency is an expression

Using webpack your module is throwing an error of Critical dependency: the request of a dependency is an expression

at the file ./node_modules/encoding/lib/iconv-loader.js 9:12-34

Can you please check?

Thanks

Warnings with webpack.

WARNING in d:/dws/projects/febs/febs/~/encoding/lib/encoding.js
Critical dependencies:
9:12-34 the request of a dependency is an expression
 @ d:/dws/projects/febs/febs/~/encoding/lib/encoding.js 9:12-34

Please dont try to fool webpack, users can use webpack.IgnorePlugin to prevent loading of 'iconv'.

Detecting charset

Any ideas on how could I detect the used charset, based on the langauge?

Just use iconv-lite?

Just wondering if it would be worth removing the node-iconv dependency in favor of iconv-lite?

test failure for ISO-2022-JP

I am packaging this module for Fedora. I have created spec file and trying to build binary rpms. when I run "nodeunit test" on Fedora 23, I get

+ nodeunit test

test
✔ General tests - From UTF-8 to Latin_1
✔ General tests - From Latin_1 to UTF-8
✔ General tests - From UTF-8 to UTF-8
✔ General tests - From Latin_13 to Latin_15
✔ General tests - From Latin_13 to Latin_15 lite
[Error: Encoding not recognized: 'ISO-2022-JP' (searched as: 'iso2022jp')]
✖ General tests - From ISO-2022-JP to UTF-8

AssertionError: <Buffer 1b 24 42 33 58 39 3b 35 3b 3d 51 30 77 38 26 3d 24 38 21 46 24 32 71 4a 73 39 70 1b 28 42> deepEqual <Buffer e5 ad a6 e6 a0 a1 e6 8a 80 e8 a1 93 e5 93 a1 e7 a0 94 e4 bf ae e6 a4 9c e8 a8 8e e4 bc 9a e5 a0 b1 e5 91 8a>
    at Object.deepEqual (/usr/lib/node_modules/nodeunit/lib/types.js:83:39)
    at Object.exports.General tests.From ISO-2022-JP to UTF-8 (/home/parag/rpmbuild/BUILD/encoding-0.1.11/test/test.js:45:14)
    at Object.<anonymous> (/usr/lib/node_modules/nodeunit/lib/core.js:236:16)
    at Object.<anonymous> (/usr/lib/node_modules/nodeunit/lib/core.js:236:16)
    at /usr/lib/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/usr/lib/node_modules/nodeunit/lib/core.js:70:9)
    at /usr/lib/node_modules/nodeunit/lib/core.js:118:25
    at /usr/lib/node_modules/async/lib/async.js:606:13
    at iterate (/usr/lib/node_modules/async/lib/async.js:134:13)
    at /usr/lib/node_modules/async/lib/async.js:145:25
    at /usr/lib/node_modules/async/lib/async.js:608:17
    at /usr/lib/node_modules/nodeunit/lib/types.js:146:17
    at process._tickCallback (node.js:442:13)


FAILURES: 1/6 assertions failed (91ms)
error: Bad exit status from /var/tmp/rpm-tmp.jrphYe (%check)

Any idea how to fix this?

about auto detect charset

this is worked fine :
var b = encoding.convert(name, 'utf8', 'gbk');
console.log(b.toString());

this not worked:
var b = encoding.convert(name, 'utf8');
console.log(b.toString());

but i must know name is gbk, is any way to detect name charset?

fails with iconv-lite unsupported encoding error

Today I ran into this when trying to parse my gmail .mbox (https://support.google.com/accounts/answer/3024195?hl=en) with the https://www.npmjs.org/package/mbox-stream module which uses https://www.npmjs.org/package/mailparser which uses encoding

[Error: Encoding not recognized: 'ISO-2022-JP' (searched as: 'iso2022jp')]

Changing useLite to false fixes the issue, but I thought I'd open an issue and ask why iconv isn't in optionalDependencies for this module?

Buffer() is deprecated

I got the message "Buffer() is deprecated" (TBH, I cannot yet reproduce it but it seems to be an issue in some browsers). It should be fixed according to

https://stackoverflow.com/questions/52165333/deprecationwarning-buffer-is-deprecated-due-to-security-and-usability-issues

Error: Cannot find module 'iconv

Error: Cannot find module 'iconv'Require stack:- C:\Program Files\nodejs\node_modules\npm\node_modules\encoding\lib\iconv-loader.js- C:\Program Files\nodejs\node_modules\npm\node_modules\encoding\lib\encoding.js- C:\Program Files\nodejs\node_modules\npm\node_modules\node-fetch-npm\src\body.js- C:\Program Files\nodejs\node_modules\npm\node_modules\node-fetch-npm\src\index.js- C:\Program Files\nodejs\node_modules\npm\node_modules\make-fetch-happen\index.js- C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\index.js- C:\Program Files\nodejs\node_modules\npm\node_modules\libnpm\fetch.js- C:\Program Files\nodejs\node_modules\npm\lib\utils\metrics.js- C:\Program Files\nodejs\node_modules\npm\lib\npm.js- C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15) at Function.Module._load (internal/modules/cjs/loader.js:746:27) at Module.require (internal/modules/cjs/loader.js:974:19) at require (internal/modules/cjs/helpers.js:92:18) at Object.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\encoding\lib\iconv-loader.js:9:13) at Module._compile (internal/modules/cjs/loader.js:1085:14) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10) at Module.load (internal/modules/cjs/loader.js:950:32) at Function.Module._load (internal/modules/cjs/loader.js:790:14) at Module.require (internal/modules/cjs/loader.js:974:19)

iconv-lite version

Hi! looks like there's no current way to resolve the new dependency ("iconv-lite": "^0.4.4") for node 0.8.x causing back-compatibility issues. Is it possible to revert the last update?