kramdown / parser-gfm Goto Github PK

Normally when you have a gem with a name like kramdown-parser-gfm, you'd expect to write require 'kramdown/parser/gfm'. Is there some reason for going against this standard?

Since this entire repository is about the GFM parser, the Kramdown::Options can be moved to a separate file.
Additionally, it'd be better to format the heredoc using Ruby 2.3 feature "squiggly heredocs" to intend the text with respect to define(...)

Trying to run the test from the cloned git repository, I get the following error:

   1) gfm codeblock fenced to html
      --- expected
      +++ actual
      @@ -16,6 +16,6 @@
       
       <p>indent with 2 spaces</p>
       
      -<div class=\"language-js highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>  <span class=\"nx\">console</span><span class=\"p\">.</span><span class=\"nx\">log</span><span class=\"p\">(</span><span class=\"s2\">\"hello\"</span><span class=\"p\">);</span>
      +<div class=\"language-js highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>  <span class=\"nx\">console</span><span class=\"p\">.</span><span class=\"nx\">log</span><span class=\"p\">(</span><span class=\"dl\">\"</span><span class=\"s2\">hello</span><span class=\"dl\">\"</span><span class=\"p\">);</span>
       </code></pre></div></div>
       "

I guess it is due to some change in the rouge highlighter between the time this test has been writtten and now. On Debian,
it fails with ruby-rouge v3.12 and later, but passes with ruby-rouge v3.1
Looking at the output, it seems to be a change in the way quotes are highlightened (before they were part of the string, now they have their own class).

Since a pipe | could mean table column in markdown, it seems that engine doesn't take into account as same as GFM does. GitHub Preview only renders table if there are joint with a heading markup, but Kramdown does also in the middle of a link or list, etc...

Then it happens thinks like reported in EbookFoundation/free-programming-books#5176

The place doesn't matter if not escaped. \| EbookFoundation/free-programming-books#6723

Example code:

* [JAVA Programming Full Project by Arabic (uni_staff project) |(المشروع الختامي بالجافا كاملا بشرح عربي )](https://www.youtube.com/playlist?list=PLnzqK5HvcpwQbsAGChtjlNPLVv6kTEXRG) - محمد شوشان

Escaped:

* [JAVA Programming Full Project by Arabic (uni_staff project) \|(المشروع الختامي بالجافا كاملا بشرح عربي )](https://www.youtube.com/playlist?list=PLnzqK5HvcpwQbsAGChtjlNPLVv6kTEXRG) - محمد شوشان

Result:

Expected:

Conclusion

So it's a bit weird without have a robust linter that detects it.

There are any attribute / config to solve it for Jekyll theme?

In GFM we can have URL's automatically turned into links without requiring any special formatting around them, yet this doesn't seem to be possible with this library. With the closest solution seeming to be:

• https://kramdown.gettalong.org/syntax.html#automatic-links

It seems this has been noticed in other projects as well, eg.:

• barryclark/jekyll-now#459 (comment)

GFM syntax allows a few more symbols to be backslash escaped:

\!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~

than it is escaped by this parser:

\\.*_+`<>()\[\]{}#!:\|"'\$=\-~

Missing ones are:

%&,/;?@\^

Is there a reason for that? It generates incorrect HTML for example\, example\@example.com.

Set up Travis CI (and optionally AppVeyor) to provide a safety net to catch bugs from future contributions.

The README documents a set of options to set, but not how to set them. Please document how to set the options or, if it is as simple as including them in the Kramdown options set, show an example.

Extra credit: show how to set options for the parser from Jekyll.

Proposal

Feel free to make these changes to the README, if they are correct, next time you update it (I do not want to submit a separate PR):

Usage:

require 'kramdown'
require 'kramdown-parser-gfm'

Kramdown::Document.new(text, input: 'GFM', gfm_quirks: ["no_auto_typographic"]).to_html

Using with Jekyll:
parser-gfm is the default in Jekyll. You do not need to specify input: "GFM" and you can pass options to it in the kramdown portion of your config file. For example:

kramdown:
   gfm_quirks: ["no_auto_typographic"]

Hi,

In order to move to rouge4 for code syntax highlighting, one test case needs to be modified: on line 16, the class of the span containing the word "log" should change from "nx" to "nf".
This change has to be coordinated with the change of the minimal version of rouge in kramdown development dependencies.

As the title says. Here is the output of running rake test (please let me know if there's anything else I can provide):

Run options: --seed 31038

# Running:

..F.F........

Finished in 0.696788s, 18.6570 runs/s, 18.6570 assertions/s.

  1) Failure:
TestFiles#test_gfm_codeblock_fenced_to_html [C:/Users/Ethan/Downloads/parser-gfm-master/test/test_files.rb:32]:
Expected: "<p>normal</p>\n\n<div class=\"language-ruby highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nb\">require</span> <span class=\"s1\">'kramdown'</span>\n\n<span class=\"no\">Kramdown</span><span class=\"o\">::</span><span class=\"no\">Document</span><span class=\"p\">.</span><span class=\"nf\">new</span><span class=\"p\">(</span><span class=\"n\">text</span><span class=\"p\">).</span><span class=\"nf\">to_html</span>\n</code></pre></div></div>\n\n<p>indent with tab</p>\n\n<pre><code>```ruby\nrequire 'kramdown'\n\nKramdown::Document.new(text).to_html\n```\n</code></pre>\n\n<p>indent with 2 spaces</p>\n\n<div class=\"language-js highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>  <span class=\"nx\">console</span><span class=\"p\">.</span><span class=\"nx\">log</span><span class=\"p\">(</span><span class=\"s2\">\"hello\"</span><span class=\"p\">);</span>\n</code></pre></div></div>\n"
  Actual: "<p>normal</p>\n\n<div class=\"language-ruby highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nb\">require</span> <span class=\"s1\">'kramdown'</span>\n\n<span class=\"no\">Kramdown</span><span class=\"o\">::</span><span class=\"no\">Document</span><span class=\"p\">.</span><span class=\"nf\">new</span><span class=\"p\">(</span><span class=\"n\">text</span><span class=\"p\">).</span><span class=\"nf\">to_html</span>\n</code></pre></div></div>\n\n<p>indent with tab</p>\n\n<pre><code>```ruby\nrequire 'kramdown'\n\nKramdown::Document.new(text).to_html\n```\n</code></pre>\n\n<p>indent with 2 spaces</p>\n\n<div class=\"language-js highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>  <span class=\"nx\">console</span><span class=\"p\">.</span><span class=\"nx\">log</span><span class=\"p\">(</span><span class=\"dl\">\"</span><span class=\"s2\">hello</span><span class=\"dl\">\"</span><span class=\"p\">);</span>\n</code></pre></div></div>\n"

  2) Failure:
TestFiles#test_gfm_header_ids_to_html [C:/Users/Ethan/Downloads/parser-gfm-master/test/test_files.rb:32]:
Expected: "<h3 id=\"myid\">test</h3>\n\n<h3 id=\"variable_name\">variable_name</h3>\n\n<h3 id=\"abc-def-öúß\">abc def öúß</h3>\n\n<h3 id=\"192-abc-192\">192 abc 192</h3>\n\n<h3>;.;;</h3>\n\n<h3 id=\"variable_name-1\">variable_name</h3>\n\n<h3 id=\"variable_name-2\">variable_name</h3>\n\n<h3 id=\"-1\">;;</h3>\n\n<h3 id=\"before--after-tab\">before \tafter tab</h3>\n\n<h3 id=\"with-code\">with <code>code</code></h3>\n\n<h3 id=\"with-äspace\">with  ä space</h3>\n\n<h3 id=\"with-smart-quotes\">With “smart” quotes</h3>\n\n<h3 id=\"with--typographic--symbols\">with — « typographic » … symbols</h3>\n\n<h3 id=\"with-m5\">with <script type=\"math/tex\">m=5</script></h3>\n"
  Actual: "<h3 id=\"myid\">test</h3>\n\n<h3 id=\"variable_name\">variable_name</h3>\n\n<h3 id=\"abc-def-öúß\">abc def öúß</h3>\n\n<h3 id=\"192-abc-192\">192 abc 192</h3>\n\n<h3>;.;;</h3>\n\n<h3 id=\"variable_name-1\">variable_name</h3>\n\n<h3 id=\"variable_name-2\">variable_name</h3>\n\n<h3 id=\"-1\">;;</h3>\n\n<h3 id=\"before--after-tab\">before \tafter tab</h3>\n\n<h3 id=\"with-code\">with <code>code</code></h3>\n\n<h3 id=\"with-äspace\">with  ä space</h3>\n\n<h3 id=\"with-smart-quotes\">With “smart” quotes</h3>\n\n<h3 id=\"with--typographic--symbols\">with — « typographic » … symbols</h3>\n\n<h3 id=\"with-m5\">with \\(m=5\\)</h3>\n"

13 runs, 13 assertions, 2 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

While building a Jekyll site we came across an issue where header IDs were being generated in non-ascii character sets. Some content is Arabic, so obviously worse there. When using the kramdown parser the transliterated_header_ids option is respected.

To reproduce it:

#!/usr/bin/env ruby
# frozen-string-literal: true

require 'kramdown'
require 'kramdown-parser-gfm'
require 'stringex'

h1 = "# nossos serviços"

kramdown_html = Kramdown::Document.new(h1, :auto_ids => true, :transliterated_header_ids => true, :input => "kramdown").to_html

gfm_html = Kramdown::Document.new(h1, :auto_ids => true, :transliterated_header_ids => true, :input => "GFM").to_html

puts "input:\n#{h1}"
puts
puts "kramdown:\n#{kramdown_html}"
puts
puts "GFM:\n#{gfm_html}"

Output:

input:
# nossos serviços

kramdown:
<h1 id="nossos-servicos">nossos serviços</h1>

GFM:
<h1 id="nossos-serviços">nossos serviços</h1>

It looks like the GFM parser is adding an id attribute to the element before the kramdown converter gets a chance to transliterate the ID.

It would be awesome if the GFM parser could be extended more easily…as it is, the initializer hard-codes span parsers (and maybe block parsers…hard to tell), so there's no way to extend without monkey-patching, or subclassing (but then anything which is configured specifically to use GFM wouldn't load the subclass).

For example, I wanted to add <mark> highlighting via :: or == which is common in many Markdown editors. I had to monkey-patch in order to do so, which isn't the end of the world, but it'd be nice to have a more extensible solution. Thoughts?

I'm using kramdown as the default markdown engine in a blog generated by Jekyll. For the most part, I'm sticking to Github-flavored Markdown (GFM) and that seems to be reasonable.

GFM advertises certain emojis as being available, but some experiments show that kramdown does not render them. In particular, I've been trying to use some flag icons for a blog post about language translation: 🇯🇵, 🇫🇷, 🇮🇳, 🇮🇹, 🇨🇳, 🇷🇺, 🇬🇷, 🇷🇴. They get rendered as text.

There's an example in a blog post here.

I realize emojis are pretty silly and may not be a priority.

Not sure if this belongs here or in gettalong/kramdown.

Very specific case here. I'm using Jekyll (default kramdown parser) and getting weird behavior with nested tables.

Input:

<table>
  <tr>
    <td>test</td>
    <td>test</td>
  </tr>
  <tr>
    <td>test</td>
    <td>
      <span>test</span>

      <table>
        <tr>
          <td>test</td>
          <td>test</td>
          <td>test</td>
        </tr>
      </table>
    </td>
  </tr>
</table>

Output:

Interestingly, in my IDE vscode, the syntax highlighting seems to think there is something wrong with this:

If I remove the newline and space between the span and the nested table, I get this also wrong and weird behavior:

If I wrap the nested stuff in a single block element...

<td>
  <div>
    <span>test</span>
    <table>
       ...
    </table>
  </div>
</td>

... or if I remove the span and only have the nested table, I get what is expected:

Btw, according to the HTML spec and validator, this is all perfectly valid HTML.

Running rake tasks get aborted with the following:

$ rake
rake aborted!
ArgumentError: Option name hard_wrap is already used
~/parser-gfm/lib/kramdown/parser/gfm.rb:16:in `<module:Options>'
~/parser-gfm/lib/kramdown/parser/gfm.rb:14:in `<module:Kramdown>'
~/parser-gfm/lib/kramdown/parser/gfm.rb:13:in `<top (required)>'
~/parser-gfm/Rakefile:7:in `require_relative'
~/parser-gfm/Rakefile:7:in `<top (required)>'
(See full trace by running task with --trace)

(Transferred from Kramdown Issue 539)

The syntax ~~content~~ does not seem to work when "content" is only one character. But the GFM actually supports it like this:

a

I'm not familiar with ruby, and fail in configuring with kramdown 2.0.0.beta2 with GFM parser. So I manually modified the test in strikethrough.text and strikethrough.html.

strikethrough.text adds ~~a~~, and strikethrough.html adds <p><del>a</del></p>, then run rake test, and I get the output:

2) Failure:
TestFiles#test_gfm_/home/tao/sandbox/parser-gfm/test/testcases/strikethrough_text_to_html [/home/tao/sandbox/parser-gfm/test/test_files.rb:32]:
--- expected
+++ actual
@@ -1,6 +1,6 @@
 "<p><del>This is a test</del></p>
 
-<p><del>a</del></p>
+<p>~~a~~</p>
 
 <p>~<del>This is another test</del>~</p>
 

It means this bug still exists in parser-gfm.

According to the Kramdown Syntax for Lists

The first list marker in a list may be indented up to three spaces. The column number of the first non-space character which appears after the list item marker on the same line specifies the indentation that has to be used for the following lines of content of the list item. If there is no such character, the indentation that needs to be used is four spaces or one tab. Indented lines may be followed by lines containing text with any amount of indentation due to line wrapping.

This means, if ordered list is (usually) used, list will take 3 spaces as the indentation for that list item.
for unordered list, it will be 2 spaces.

I'm wondering would it be possible for GFM (since it is default preprocessor for Jekyll - therefore github pages) to be matched with Github style? (consistent 4 spaces) Or at least provide option to it?

So for example, below markdown will print proper codeblocks with NO SPACE at the start of the codeblock.
But for the Kramdown with GFM preprocessor, codeblocks WILL HAVE unintended spaces at the start of the codeblock
(and I'm delibrately using 1. to show that they are from the same list)

Markdown:

1. First ordered item

    ```default
    Codeblock with one tab(four space) indentation
    ```

1. Second ordered item

   ```default
   Codeblock with **3 spaces** indentation
   ```

1. Third ordered item

* First unordered item

    ```default
    Codeblock with one tab(four space) indentation
    ```

* Second unordered item

  ```default
  Codeblock with **two spaces** indentation
  ```

* Third unordered item

Github Example: No space at the start of the codeblock

1. First ordered item

   Codeblock with one tab(four space) indentation
   

2. Second ordered item

   Codeblock with **3 spaces** indentation
   

3. Third ordered item

• First unordered item

  Codeblock with one tab(four space) indentation
  

• Second unordered item

  Codeblock with **two spaces** indentation
  

• Third unordered item

Jekyll Blog Example:

• Screenshot

• html

<ol>
  <li>
    <p>First ordered item</p>

    <pre><code class="language-default"> Codeblock with one tab(four space) indentation
</code></pre>
  </li>
  <li>
    <p>Second ordered item</p>

    <pre><code class="language-default">Codeblock with **3 spaces** indentation
</code></pre>
  </li>
  <li>
    <p>Third ordered item</p>
  </li>
</ol>

<ul>
  <li>
    <p>First unordered item</p>

    <pre><code class="language-default">  Codeblock with one tab(four space) indentation
</code></pre>
  </li>
  <li>
    <p>Second unordered item</p>

    <pre><code class="language-default">Codeblock with **two spaces** indentation
</code></pre>
  </li>
  <li>
    <p>Third unordered item</p>
  </li>
</ul>

Thank you.

Trying to run the tests from a fresh cloned repository, I get the following error:

 gfm header ids to html
      --- expected
      +++ actual
      @@ -24,5 +24,5 @@
       
       <h3 id=\"with--typographic--symbols\">with — « typographic » … symbols</h3>
       
      -<h3 id=\"with-m5\">with <script type=\"math/tex\">m=5</script></h3>
      +<h3 id=\"with-m5\">with \\(m=5\\)</h3>
       "

I could not find the part of kramdown which is supposed to create the <script type="math/tex"> part
I am using kramdown v2.3 on Debian.

When looking at RubyGems there was a release of version 1.1.0 on 2019-05-29.

This release is not reflected on GitHub: Neither is there a 1_1_0 tag nor is the version updated in file lib/kramdown/parser/gfm.rb.

Could you please add missing code changes and make a proper source release? In that way, the newest release can be added to Free Software distributions, like GNU Guix.

Thank you.

Here are two examples with their expected output:

Input:

This ~~is a complex *strike* through *test ~~with nesting~~ involved* here~~.

Output:

<p>This <del>is a complex <em>strike</em> through <em>test <del>with nesting</del> involved</em> here</del>.</p>

Input:

This ~~is a complex *strike* through *test~~ without nesting ~~involved* here~~.

Output:

<p>This <del>is a complex <em>strike</em> through *test</del> without nesting <del>involved* here</del>.</p>

Current implementation:

STRIKETHROUGH_DELIM = /~~/.freeze
STRIKETHROUGH_MATCH = /#{STRIKETHROUGH_DELIM}(?!\s|~).*?[^\s~]#{STRIKETHROUGH_DELIM}/m.freeze

Is able to successfully perform the first, but not the second case.

Here is an example running the two on commonmark:

• This is a complex strike through test with nesting involved here.
• This is a complex strike through *test without nesting involved* here.

I believe I'm on the right direction here, but so far what I've manage to do was get the second example to work by breaking the first one.

What I did was base on the balanced parameters example, using some lookahead and lookbehind code:

STRIKETHROUGH_MATCH = /(?<re>~~(?!\s|~)((?:.*)|\g<re>)?(?<!\s|~)~~)/m.freeze

This is a clone of the issue originally reported here: gettalong/kramdown#538

This block of Markdown

[eeny] [meeny] [miny] [moe]

[eeny]: http://example.com
[meeny]: http://example.com
[miny]: http://example.com
[moe]: http://example.com

renders all four links here on GitHub as well as on CommonMark's parser:

eeny meeny miny moe

However, Kramdown renders it as:

eeny miny

I don't know if the Kramdown project is interested in tracking GitHub or CommonMark's behavior that closely, but at least from a naive user perspective, I think their behavior is more intuitive than Kramdown's.

Migrated from gettalong/kramdown#733.

echo '```python\nfoo()\n```' | \
kramdown \
  --extension=parser-gfm \
  --syntax-highlighter=rouge \
  --syntax-highlighter-opts "{}"

Result:

<p><code>python
foo()
</code></p>

echo '~~~python\nfoo()\n~~~' | \
kramdown \
  --extension=parser-gfm \
  --syntax-highlighter=rouge \
  --syntax-highlighter-opts "{}"

Result:

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">foo</span><span class="p">()</span>
</code></pre></div></div>

I would recommend the use of older rocket syntax in hashes because:

• it mirrors the output from Ruby

  gamma = { foo: 'bar' }
  delta = { :foo => 'bar' }
  
  puts gamma  # => {:foo=>"bar"}
  puts delta  # => {:foo=>"bar"}

• it allows aligning keys for improving readability

  a = {
    :name    => 'John Doe',
    :age     => 35,
    :address => '122B, Baker Street'
  }
  
  b = {
    name: 'John Doe',  # name   : 'John Doe' would raise a syntax error
    age: 35,
    address: '122B, Baker Street'
  }

• Does not resemble JSON:

  payload = {
    "name": "John Doe",
    "age": "35",
    "address": "122B, Baker Street"
  }
  
  puts payload # => {:name=>"John Doe", :age=>"35", :address=>"122B, Baker Street"}

For the following code snippet:

<div dir="rtl">
## توافقنامه‌ی مجوز همکاری

مشارکت در این مخزن به معنی موافقت شما با مجوز [LICENSE](/blob/master/LICENSE) این مخزن است.
</div>

The expected HTML to be rendered is:

<div dir="rtl">
    <h2>توافقنامه‌ی مجوز همکاری</h2>
    <p>مشارکت در این مخزن به معنی موافقت شما با مجوز <a href="blob/master/LICENSE">LICENSE</a> این مخزن است.</p>
</div>

And GFM renders that perfectly:
https://github.com/EbookFoundation/free-programming-books/blob/e629db6b05a72fd64a6b33feeeebbe830641cf10/CONTRIBUTING-fa_IR.md

However kramdown-parser-gfm appears to stop processing any markdown once it is nested under an HTML tag
https://ebookfoundation.github.io/free-programming-books/CONTRIBUTING-fa_IR.html

See EbookFoundation/free-programming-books#5559 for more information

Please forgive me if this is intended behavior, perhaps following a spec somewhere, but it seems that kramdown will autogenerate IDs that start with a number, but it will not respect manually specifying IDs. I don't see my usecase disclaimed under Specifying a Header ID. I'm cross-posting from gettalong/kramdown#711, where @gettalong suggested that this was expected behavior using the kramdown binary, but I'm still not 100% sure why, particularly because of the automatic generation of header IDs and how that differs from the behavior using the binary.

Autogeneration of ID for headings starting with numbers works

For example:

## 2021-03-30

yields

<h2 id="2021-03-30">2021-03-30</h2>

as expected. (The example remains true for header text that is purely numeric, such as ## 123.)

(Although, I didn't understand how to reconcile that with the HTML Converter documentation for "Automatic Generation of Header IDs", which suggested that I should expect a different result.)

Specifying a custom header ID works for headings starting with [a-z]

And

## Foo
{: #bar}

and

## Foo {#bar}

both yield

<h2 id="bar">Foo</h2>

also as expected.

Specifying a custom header ID starting with [0-9] does not work

So here's where I get the unexpected behavior:

## 2021-03-30: Foo {#2021-03-30}

yields

<h2 id="2021-03-30-foo-2021-03-30">2021-03-30: Foo {#2021-03-30}</h2>

and

## 2021-03-30: Foo
{: #2021-03-30}

yields

<h2 id="2021-03-30-foo">2021-03-30: Foo</h2>

Of course, I was expecting the following in both of the two examples immediately above:

<h2 id="2021-03-30">2021-03-30: Foo</h2>

I'm gathering that kramdown is ignoring manually specified header IDs when they start with numbers (since even adding [a-z] characters later in the ID doesn't help).

Context

I'm using GitHub Pages, and my Gemfile.lock file specifies the following under github-pages (213):

kramdown (= 2.3.0)
kramdown-parser-gfm (= 1.1.0)

And under jekyll (3.9.0):

kramdown (>= 1.17, < 3)

and then also

    kramdown (2.3.0)
      rexml
    kramdown-parser-gfm (1.1.0)
      kramdown (~> 2.0)

My _config.yml includes the following lines:

kramdown:
  smart_quotes: ["apos", "apos", "quot", "quot"]
  typographic_symbols: { hellip: ... , mdash: --- , ndash: -- , laquo: "<<" , raquo: ">>" , laquo_space: "<< " , raquo_space: " >>" }
  auto_id_stripping: true

I noticed that I didn't have a markdown: kramdown specified, but adding that in didn't change the behavior, so I'm thinking that maybe that becomes implicit if there's a kramdown object specified.

FWIW, I've also confirmed the same behavior on this online kramdown editor/renderer, so it doesn't seem to be specific to a GitHub Pages or Jekyll implementation.

It's a lot easier to see effects immediately if you include a TOC block:

* TOC
{:toc}

I found it a bit surprising that the gemspec was embedded in Rakefile rather than tracked as a separate file. Would it be possible/acceptable to extract it out? It would make it easier for new developers I think, as well as useful for gem users to see this history of this specific file.