Giter Club home page Giter Club logo

i18n-extract's Introduction

i18n-extract

Manage localization with static analysis.

npm version npm downloads Build Status

Dependencies DevDependencies

Installation

npm install --save-dev i18n-extract

The problem solved

This module analyses code statically for key usages, such as i18n.t('some.key'), in order to:

  • Report keys that are missing
  • Report keys that are unused.
  • Report keys that are highly duplicated.

E.g. This module works well in conjunction with:

Supported keys

  • static:
i18n('key.static')
  • string concatenation:
i18n('key.' + 'concat')
  • template string:
i18n(`key.template`)
  • dynamic:
i18n(`key.${dynamic}`)
  • comment:
/* i18n-extract key.comment */

API

extractFromCode(code, [options])

Parse the code to extract the argument of calls of i18n(key).

  • code should be a string.
  • Return an array containing keys used.
Example
import {extractFromCode} from 'i18n-extract';
const keys = extractFromCode("const followMe = i18n('b2b.follow');", {
  marker: 'i18n',
});
// keys = ['b2b.follow']

extractFromFiles(files, [options])

Parse the files to extract the argument of calls of i18n(key).

  • files can be either an array of strings or a string. You can also use a glob.
  • Return an array containing keys used in the source code.
Example
import {extractFromFiles} from 'i18n-extract';
const keys = extractFromFiles([
  '*.jsx',
  '*.js',
], {
  marker: 'i18n',
});

Options

  • marker: The name of the internationalized string marker function. Defaults to i18n.
  • keyLoc: An integer indicating the position of the key in the arguments. Defaults to 0. Negative numbers, e.g., -1, indicate a position relative to the end of the argument list.
  • parser: Enum indicate the parser to use, can be typescript or flow. Defaults to flow.
  • babelOptions: A Babel configuration object to allow applying custom transformations or plugins before scanning for i18n keys. Defaults to a config with all babylon plugins enabled.

findMissing(locale, keysUsed)

Report the missing keys. Those keys should probably be translated.

  • locale should be a object containing the translations.
  • keysUsed should be an array. Containes the keys used in the source code. It can be retrieve with extractFromFiles our extractFromCode.
  • Return a report.
Example
import {findMissing} from 'i18n-extract';
const missing = findMissing({
  key1: 'key 1',
}, ['key1', 'key2']);

/**
 * missing = [{
 *   type: 'MISSING',
 *   key: 'key2',
 * }];
 */

Plugins

findUnused(locale, keysUsed)

Report the unused key. Those keys should probably be removed.

  • locale should be a object containing the translations.
  • keysUsed should be an array. Containes the keys used in the source code. It can be retrieve with extractFromFiles our extractFromCode.
  • Return a report.
Example
import {findUnused} from 'i18n-extract';
const unused = findUnused({
  key1: 'key 1',
  key2: 'key 2',
}, ['key1']);

/**
 * unused = [{
 *   type: 'UNUSED',
 *   key: 'key2',
 * }];
 */

findDuplicated(locale, keysUsed, options)

Report the duplicated key. Those keys should probably be mutualized. The default threshold is 1, it will report any duplicated translations.

  • locale should be a object containing the translations.
  • keysUsed should be an array. Containes the keys used in the source code. It can be retrieve with extractFromFiles our extractFromCode.
  • options should be an object. You can provide a threshold property to change the number of duplicated value before it's added to the report.
  • Return a report.
Example
import {findDuplicated} from 'i18n-extract';
const duplicated = findDuplicated({
  key1: 'Key 1',
  key2: 'Key 2',
  key3: 'Key 2',
});

/**
 * unused = [{
 *   type: 'DUPLICATED',
 *   keys: [
 *     'key2',
 *     'key3',
 *   ],
 *   value: 'Key 2',
 * }];
 */

forbidDynamic(locale, keysUsed)

Report any dynamic key. It's arguably more dangerous to use dynamic key. They may break.

  • locale should be a object containing the translations.
  • keysUsed should be an array. Containes the keys used in the source code. It can be retrieve with extractFromFiles our extractFromCode.
  • Return a report.
Example
import {forbidDynamic} from 'i18n-extract';
const forbidDynamic = forbidDynamic({}, ['key.*']);

/**
 * forbidDynamic = [{
 *   type: 'FORBID_DYNAMIC',
 *   key: 'key.*',
 * }];
 */

flatten(object)

Flatten the object.

  • object should be a object.
Example
import {flatten} from 'i18n-extract';
const flattened = flatten({
  key2: 'Key 2',
  key4: {
    key41: 'Key 4.1',
    key42: {
      key421: 'Key 4.2.1',
    },
  },
});

/**
 * flattened = {
 *   key2: 'Key 2',
 *   'key4.key41': 'Key 4.1',
 *   'key4.key42.key421': 'Key 4.2.1',
 * };
 */

mergeMessagesWithPO(messages, poInput, poOutput)

Output a new po file with only the messages present in messages. If a message is already present in the poInput, we keep the translation. If a message is not present, we add a new empty translation.

  • messages should be an array.
  • poInput should be a string.
  • poOutput should be a string.
Example
import {mergeMessagesWithPO} from 'i18n-extract';

const messages = ['Message 1', 'Message 2'];
mergeMessagesWithPO(messages, 'messages.po', 'messages.output.po');

/**
 * Will output :
 * > messages.output.po has 812 messages.
 * > We have added 7 messages.
 * > We have removed 3 messages.
 */

License

MIT

i18n-extract's People

Contributors

brandonbloom avatar ctavan avatar fajzen avatar gitter-badger avatar greenkeeperio-bot avatar gregberge avatar gsouf avatar jamtat avatar kerumen avatar kmaschta avatar magicmark avatar mathieuletyrant avatar mxmul avatar oliviertassinari avatar oreqizer avatar phyks avatar sheerun avatar v-karbovnichy avatar xiao-hu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

i18n-extract's Issues

`findUnused` doesn't work with fully dynamic keys

I have this

const title = condition ? 'key:one' : 'key:two'
return t(title)

This is parsed as a dynamic key ('*') and catch all the keys for the findUnused method. Hence the findUnused returns nothing although I do have unused keys.

Can I propose a PR to exclude "fully" dynamic keys in the findUnused method?

Finding JSX React props

Would be awesome to add support for Component values in react. react-18next for example you specify the key as a prop.

<I18N i18n={'someKey'} />

Would that be possible to support, I haven't dug into the code just yet.

Error while using findUnused() and findMissing()

Tried the same example as GitHub-

const unused = findUnused({
key1: 'key 1',
key2: 'key 2',
}, ['key1']);

/node_modules/i18n-extract/lib/findUnused.js:18
if (keyUsed.includes('*')) {
^

TypeError: Cannot read property 'includes' of undefined

Error in 0.6.4: The 'decorators' plugin requires a 'decoratorsBeforeExport' option

I get this error when trying to update to the latest version.

The 'decorators' plugin requires a 'decoratorsBeforeExport' option, whose value must be a boolean. If you are migrating from Babylon/Babel 6 or want to use the old decorators proposal, you should use the 'decorators-legacy' plugin instead of 'decorators'.

Iโ€™ve tried changing the babel config on my end but that has not been successful.

Be able to ignore some lines of code through a comment

Hi,

It seems i18n-extract has the ability to include some translation strings through comments, with

/* i18n-extract key.comment */

However, I could not find a way to ignore a given line of code with a comment (similar to what eslint does). This could be useful as sometimes you really need a dynamic key but it can take a very limited range of values that you know in advance. You could ignore the dynamic key with a comment, and eventually add extra comments to include all the possible values.

This would really be awesome :)

Thanks!

Unsupported node

We should provide a better support for code like:

const boolToStr = a => a ? 'yes' : 'no';
i18n.t('account.appointments.details.handicap_access', {yesno: i18n.t(boolToStr(handicap))});

i18n.t(`languages.${language.iso_code}`);

Support multiple arguments

Hello, I'd like to ask why doesn't this plugin allow multiple arguments in the translation function? I need this feature for i18next's interpolation feature:

i18n("Hello, {{username}}!", {username: user.username});

To me, disallowing such messages seems pointless but maybe there is some rationale behind it?
I can submit a PR once we agree how to handle multiple arguments

Webpack integration

Hello,

Is there a recommended way to integrate this with a webpack build? Otherwise, would this be a good candidate for a plugin?

Thanks!

support ES6

Would be great to support full ES6 feature.
I'm using facebook fork of esprima. This fork has an Experimental support for ES6/Harmony (module, class, destructuring, ...). Don't know if it's enough.

mergeMessagesWithPO fails for absolute paths

I've been using [email protected] so far. Today I upgraded to 0.4.1 and encountered a strange problem...

Here's how I use i18n-extract. I collect messages and update my translation files

function collectMessages() {
    const jsFiles = path.join(__dirname, 'assets/js/**/*.js')
    const messages = i18nExtract.extractFromFiles(jsFiles, { marker: '__' })

    languages.forEach((lang) => {
        const out = translationFilePath(lang)
        <b>i18nExtract.mergeMessagesWithPO(messages, out, out)</b>
    })
}

translationsFilePath is a function that returns an absolute path. However, this absolute path gets then prepended with /path/to/i18n-extract/lib/.

This seems to be caused by path.join(__dirname, ...) in i18n-extract's code. See the path.join call on this line

I think it's a flaw in the design of node's path.join, it seems to naively join everything it receives. Neverthe less, this probably should be fixed - maybe check if the path is relative or absolute first?

Exclude in paths

It will be nice use some exclude/ignore pattern for paths or external resolver like anymatch

Get code fragments with keys

Hi,

It does not seem to be possible to get code fragments from which the keys were matched. I use extractFromFiles to extract i18n keys from my code and check whether all my translations are actually used and so on.

However, as I added it on a previous code base, I have a lot of jokers returned. Typically, when I pass a variable to my translator, I get a * joker in the output keys from i18n-extract.

I am trying to rewrite my code to narrow down these jokers, but it is quite painful to find where it comes from in the code. It would be a really great addition to return, besides the matched keys, the file where it is coming from and the code fragment which gave the match. I think all this info is already available to i18n-extract, so I may have missed something to get it from the library. :/

Thanks!

EDIT: This would be especially useful since forbidDynamic could give extra information on the code fragments to fix :)

expose `extractFromAST` API?

Hey โ€“ thanks for maintaining this project!

My team has built some tooling that runs a few Babel transforms on a source file before passing it to i18n-extract. Right now that means parsing the source file, applying our transformations on the AST, getting the transformed code, then passing to extractFromCode, which parses everything again. This double parsing can be pretty slow when applied to a large number of files, and I'd like some way to avoid it.

What do you think about exporting extractFromAST from this library? I can make a PR if you're okay with the approach

[BABEL] unknown: Preset /* your preset */ requires a filename to be set when babel is called directly,

Hello,

I've been getting this error in my React Native project

Code:

    usedKeys = extractor.extractFromFiles(['./src/**/*.js'], {
      marker: 't',
    });

Error:

    [BABEL] unknown: Preset /* your preset */ requires a filename to be set when babel is called directly,
    ```
    babel.transform(code, { filename: 'file.ts', presets: [/* your preset */] });
    ```
    See https://babeljs.io/docs/en/options#filename for more information.

      at validateIfOptionNeedsFilename (node_modules/i18n-extract/node_modules/@babel/core/lib/config/full.js:274:11)
      at node_modules/i18n-extract/node_modules/@babel/core/lib/config/full.js:286:52
          at Array.forEach (<anonymous>)
      at validatePreset (node_modules/i18n-extract/node_modules/@babel/core/lib/config/full.js:286:25)
      at loadPresetDescriptor (node_modules/i18n-extract/node_modules/@babel/core/lib/config/full.js:293:3)
          at loadPresetDescriptor.next (<anonymous>)
      at recurseDescriptors (node_modules/i18n-extract/node_modules/@babel/core/lib/config/full.js:107:30)
          at recurseDescriptors.next (<anonymous>)
      at loadFullConfig (node_modules/i18n-extract/node_modules/@babel/core/lib/config/full.js:142:6)
          at loadFullConfig.next (<anonymous>)

babel.config.js:

module.exports = {
  presets: ['module:metro-react-native-babel-preset'],
};

Downgrading i18n-extract to 0.6.3 fixes it so this isn't really urgent, but it would be extremely nice to get this fixed :)

Thanks again for the great library!

`extractFromFiles` is not working with scopes

if your file contains

i18n.t(patient.import_identifier ? 'with_import_identifier' : 'without_import_identifier',
      { scope: 'booking.patient.format.fullname.long' })

the
extractFromFiles will returns:

[ { key: 'with_import_identifier',
    loc: SourceLocation { start: [Position], end: [Position],
    file: '.../PatientVerificationCard.js' },
  { key: 'without_import_identifier',
    loc: SourceLocation { start: [Position], end: [Position] },
    file:' .../PatientVerificationCard.js' } ]

Is MR still supported ?

New feature:
Scan all strings, intersect with translations provided.

code:

// Provider
<React.Context value={{ titleKey: 'a' }} />


// Consumer
const { titleKey } = useContext();
return t(titleKey)
const translations = ['a', 'b', 'c',]
scan(`code...`, translations) // output:  'a'

Make distinction between keys with a star in the text and dynamic keys

Dynamic keys will be returned as a string with a star inside ti represent the unknown part. We cannot distinct them from string with a star inside. Can we make this more explicite, for example having a flag in the returned object saying that this is a dynamic string.

Or if it's too much work can we have an option to choose a custom token to replace the star character?

Whatever suits you, I'll open a PR

Using extractFromFiles with Swig, Nunjucks or Twig templates

I am trying to extract keys using extractFromFiles with a templating engine with this module, a simple example of a template:

`


{i18n('hello')}

{{i18n('morning')}}

`

In all cases, double curly brackets, emits an error because it thinks the templates are JSX files
/Research/i18n_extract_test/node_modules/babylon/lib/index.js:7333 throw jsxError || err;
the key is extracted from the translation with single curly braces no matter what the file endings are. How can I configure the module to recognise file types?
Thanks

babel.config.js vs .babelrc

We recently switched to a babel.config.js config instead of a .babelrc.js config, we're getting the following error:

(node:16257) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'end' of undefined
    at CallExpression (/Users/peterp/Chatterbug/chatterbug/node_modules/i18n-extract/lib/extractFromCode.js:166:42)
    at NodePath._call (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/path/context.js:53:20)
    at NodePath.call (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/path/context.js:40:17)
    at NodePath.visit (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/path/context.js:88:12)
    at TraversalContext.visitQueue (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/context.js:118:16)
    at TraversalContext.visitSingle (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/context.js:90:19)
    at TraversalContext.visit (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/context.js:146:19)
    at Function.traverse.node (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/index.js:94:17)
    at NodePath.visit (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/path/context.js:95:18)
    at TraversalContext.visitQueue (/Users/peterp/Chatterbug/chatterbug/node_modules/@babel/traverse/lib/context.js:118:16)
(node:16257) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:16257) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

It seems like our code is no longer transpiled by babel, switching the config back to .babelrc.js fixed the problem.

Is there something that I'm misunderstanding that's causing this?

Does not seem to play nice with .vue files.

Hi all,

I've been using polyglot with Vue as a Vue mixin, and tried using this tool to analyze .vue components as well as plain .js modules. It works wonderfully if I am only using it on the JavaScript files, but adding a Vue component to the mix results in the following error:

throw jsxError || err;
          ^

SyntaxError: Unexpected token (3:46)
    at Parser.pp$5.raise (/Users/seanohue/node_modules/babylon/lib/index.js:4454:13)
    at Parser.pp.unexpected

Note that I am not using JSX with Vue. It seems to be attempting to parse the Vue files as though they are JSX, and fails when running into... something unexpected. Note that it also does not tell me where in the files it was trying to parse that it failed on, so I am not sure which token it found issue with.

Filename missing when SyntaxError thrown

I extract from .flow.js files, and babel parses these quite well. But if I make a syntax error, the output isnt very helpful (see below). Could we catch the SyntaxError here and perhaps re-throw it with a filename?

/node_modules/i18n-extract/node_modules/babylon/lib/index.js:788
    throw err;
    ^

SyntaxError: Unexpected token, expected "," (27:17)
    at _class.raise (.../node_modules/i18n-extract/node_modules/babylon/lib/index.js:776:15)
...
    at _class.parse (.../node_modules/i18n-extract/node_modules/babylon/lib/index.js:5304:17)
    at parse (.../node_modules/i18n-extract/node_modules/babylon/lib/index.js:10095:38)
    at extractFromCode (.../node_modules/i18n-extract/lib/extractFromCode.js:79:32)
    at .../node_modules/i18n-extract/lib/extractFromFiles.js:30:54
    at Array.forEach (<anonymous>)
    at extractFromFiles (.../node_modules/i18n-extract/lib/extractFromFiles.js:27:10)
    at bootstrap_node.js:609:3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.