Proposal to use mdit-py-i18n (along with markdown-it-py) for Markdown support about translate HOT 7 OPEN

For LibreOffice, having support for Hugo static site generator would be nice to have indeed as simple Markdown support is not enough for our website. Of course it is a big ask to change the Markdown library when support was just recently implemented and I understand if the idea is rejected.

from translate.

• mistletoe is CommonMark-compliant as well
• line numbers are indeed missing right now, but it seems close, see miyuchina/mistletoe#188
• mistletoe can be extended as well, which of them can be extended easier or has more plugins is a more complex topic and I have no knowledge here

I can't speak for @anderskaplan as author of this integration, but I don't really see a compelling reason for switch.

@ilmari-lauhakangas https://github.com/PhuNH/mdit-py-hugo adds just support for additional Hugo syntax, that doesn't sound like a site generator you mention. If you are missing some feature, please open a separate issue for that.

PS: Some insights on choosing the parser is in #3956

from translate.

I would like to make it clearer that the important difference is not one between the parsers themselves, but the number of syntaxes with i18n support already implemented (based on those parsers). It's true that Translate's Markdown support is currently based on mistletoe, just like mdit-py-i18n is based on markdown-it-py, but:

• So far it seems to me that Translate has support for Markdown table (i.e. it's not about mistletoe being able to parse table syntax and generate HTML, but about the fact that strings can be extracted from a Markdown table and then a table in other languages can be generated), but not for definition list and front matter. mdit-py-i18n already works with those;
• I forgot to mention that mdit-py-i18n can process single comments in fence code blocks too. Currently // and # comments are supported;
• mdit-py-hugo adds support for additional Hugo syntaxes, that's true. And that's the basis for the i18n support for those syntaxes, which can be added to Translate with just some small adjustments I suppose (if mdit-py-i18n is used).

Of course you can say that the support for those can also be added to Translate while still being based on mistletoe, however that's works which need to be done, while with mdit-py-i18n, they are done already. And with mdit-py-i18n, you have something that's already used and tested with KDE's various Markdown files, while, for example the line number thingy, you don't even have it yet.

I hope that makes it clearer.

from translate.

Of course you can say that the support for those can also be added to Translate while still being based on mistletoe, however that's works which need to be done, while with mdit-py-i18n, they are done already.

Well, using mdit-py-i18n inside the translate-toolkit would also be a work which needs to be done :-).

The problem with Markdown lies in the extensions and conventions.

You needed definition list extension and bundled it with mdit-py-i18n, what is certainly good for your use case, but might cause difficulties if the user is not using it. Okay, starting a paragraph with : is not that likely, but in general, the undesired extensions can cause issues. The more you enable them, the more likely it is to cause unexpected behavior.

On the other side, both mdit-py-i18n and translate-toolkit skip code blocks, while some users might want to translate these, see #5048.

Anyway, I don't care which parser is used underneath. If mdit-py-i18n was available while discussing #3956, it would be definitely considered back then, but it was not.

from translate.

Well, using mdit-py-i18n inside the translate-toolkit would also be a work which needs to be done :-).

A much smaller work, I would say, because I have all the building blocks ready already.

The ability to choose which additional syntaxes to enable is also something I plan to support. There being a handler for a type of syntax in mdit-py-i18n doesn't necessarily mean that its users have to follow the syntax every time.

As for code blocks, I mentioned the support for comments in fence code blocks exactly because I know about that issue. Here I partly have the same opinion as Anders, that code blocks should not always be internationalized. Having all code blocks added into PO/XLIFF files is like just splitting Markdown files into paragraphs and adding all those paragraphs into PO/XLIFF, there's no need to parse the file content, they are all code anyway. So with mdit-py-i18n, if you use an indented code block, you are assumed to want the block not translated. In case you use a fenced code block (and in the best case specify what language the code is in), the tool has, or will try to figure out, the needed information and can help you with parsing it. I think that's a fair expectation.

Anyway, that's my proposal. Feel free to close this when you want.

from translate.

In my experience, Markdown is much more complicated than one would believe from just looking at the syntax. And the parsing is only a part of it -- the challenge is to make a clean roundtrip from Markdown to some intermediate, translatable format which fits inside the Translate toolkit, and then back to Markdown again. All while handling things such as text with emphasis in a table within a list wrapped in a block quote. :)

I have nothing against the proposal, as long as it doesn't make Markdown translation in the Translate toolkit worse. Just be aware that it might be (a lot) easier to work with what is already there and try to extend it. Mistletoe is extensible and accepts contributions, too.

from translate.

@PhuNH I think you're underestimating the effort of reimplement current code with mdit-py-i18n. But if you are interested in that, we can give it a try.

From a quick test I did now, there is a major difference between the two implementations in handling inline HTML and autolinks. The current implementation does not expose these for translation, for example:

This is almost the same question as the code blocks mentioned before. In some situations, you want to translate these (for example links to localized websites), while in others you want to avoid translators to mess up these.

from translate.