This LUA filter stopped working with the format `commonmark_x`, and is not displayed correctly with the format `markdown_github` on Codebraid about list-table HOT 31 CLOSED

In the commonmark_x case, the issue is that the Lua filter isn't compatible with the divs that Codebraid Preview inserts into the abstract syntax tree to provide scroll sync. Scroll sync for commonmark_x is different than for all other formats, because Pandoc provides source location information instead of Codebraid Preview having to attempt to reconstruct it. There's a note in the README about adapting filters. Basically, they need to skip elements with a data-pos attribute.

In the markdown_github case, it appears that the issue is that the format doesn't automatically support fenced divs. So you need to specify the input format as markdown_github+fenced_divs.

from list-table.

In the markdown_github case, it appears that the issue is that the format doesn't automatically support fenced divs. So you need to specify the input format as markdown_github+fenced_divs.

I adjusted the confluence.yaml, updating from to markdown_github+fenced_divs, but Codebraid Preview informed the failure. The problem is probably in the Lua list-table code.

from list-table.

It seems to be working fine for me. I'd suggest checking the Markdown document between the sh code block and whatever follows. For some reason, this is being treated as code instead of regular Markdown. There might be some sort of syntax error in the document. Maybe the * in the list are indented too far.

from list-table.

It seems to be working fine for me. I'd suggest checking the Markdown document between the sh code block and whatever follows. For some reason, this is being treated as code instead of regular Markdown. There might be some sort of syntax error in the document. Maybe the * in the list are indented too far.

• I removed the sh part;
• I fixed the * indentation;

But still the same error. I investigated the table indentation. I fixed, using VSCode's Ctrl + [, and it worked now:

:::list-table
Markup languages

* - Name
  - Initial release

* - Markdown
  - 2004

* - reStructuredText
  - 2002
:::

in comparison to the buggy code copied from this repository's README.md:

:::list-table
   Markup languages

   * - Name
     - Initial release

   * - Markdown
     - 2004

   * - reStructuredText
     - 2002
:::

from list-table.

Only the complicated tables do not work.

from list-table.

Ah, I reviewed my old Codebraid projects, I copied all the missing features for markdown_github variant format and all the complicated tables worked now.

The @pandoc-ext personnel to document the required features for any list-table table to work. They need to fix the table indentation bug.

from list-table.

Here are the missing features:

from: markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum

from list-table.

from list-table.

@wlupton

Please could you supply the full command line that shows the problem with the example below? Thanks.

With 3 indentation spaces:

• Using the markdown_github with enabled on Codebraid Preview, it failed.
• Using the full command pandoc --preserve-tabs --standalone --wrap=auto --defaults "confluence.yaml" --from markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum --to html "README.md", it also failed.

Switching to commonmark_x.

• Using the Codebraid Preview, and disabling the scroll settings, it failed.
• Using the full command pandoc --preserve-tabs --standalone --wrap=auto --defaults "confluence.yaml" --from commonmark_x+bracketed_spans+fenced_divs+raw_attribute+raw_html --to html "README.md" -o "result.html", it worked.

from list-table.

Thanks for that @gusbemacbe (I realize that I should have been able to extract this info from the existing discussion; sorry about that).

I think @gpoore's analysis is spot on, and that the main thing is that you need to enable the fenced_divs extension. With this input:

% cat spiz.md
:::list-table
- - zero-indented
:::

:::list-table
   - - three-indented
:::

...this works for me:

% pandoc spiz.md -f markdown_github+fenced_divs -L list-table.lua 
[WARNING] Deprecated: markdown_github. Use gfm instead.
<table>
<thead>
<tr class="header">
<th>zero-indented</th>
</tr>
</thead>

</table>
<table>
<thead>
<tr class="header">
<th>three-indented</th>
</tr>
</thead>

</table>

So why do you get the original codebraid error? I suspect (from the output shown below) that the indentation has somehow been changed. Is it possible to show precisely the markdown that's being passed to pandoc?

As for possible updates to the filter, I think that we could/should emphasise (in the documentation) that the fenced_div extension has to be enabled. Also, I suppose that if any of the supported pandoc markdown variants don't support 3-space indentation (are there any that don't?) then we could remove it from the examples. I also think that the filter should be adapted to work with codebraid (I can create a separate issue for that).

from list-table.

Here is the full command given by Codebraid:

pandoc "README.md"
	--css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/%40vscode/codicons/dist/codicon.css" \ 
	--css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/vscode-markdown.css" \
	--css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/codebraid-preview.css" \
	--preserve-tabs --standalone --wrap=auto \ 
	--defaults "$HOME/Tests/confluence/_codebraid/temp_defaults/_codebraid_preview.yaml" \
	--lua-filter="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/sourcepos_sync.lua" \
	--lua-filter="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/show_raw.lua" \
	--katex="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/katex/dist/" \ 
	--from markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum \
	--to html

In the case of defaults, replace the path with where the defaults file is located. Copy my defaults code:

from: markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum
default-image-extension: '.png'
extract-media: dir
metadata:
  author: "Gustavo Costa"
  date: '2023-12-29'
  keywords:
    - codebraid
    - css
    - guide
    - html
    - markdown
    - markdown_mmd
    - multimarkdown
    - pandoc
  lang: "pt-PT"
  title: "Codebraid Pandoc Markdowm Guide"
  toc-title: "Table of Guides"
  version: "1.0.0"
preserve-tabs: true
shift-heading-level-by: -1
tab-stop: 2
trace: true
track-changes: accept

# General writer options
filters:
  - "list-table.lua"
highlight-style: kate
no-check-certificate: true
request-headers:
  - ["User-Agent", "Mozilla/5.0"]
strip-comments: false
toc: true
toc-depth: 7
variables:
  confluence_html: true
  languages:
    - english
    - español
    - français
    - português

# Options affecting specific writers
ascii: false
email-obfuscation: references
html-q-tags: true
incremental: true
listings: true
markdown-headings: atx
number-sections: true
reference-links: true
reference-location: block
section-divs: true
self-contained: false
slide-level: 2
top-level-division: chapter

# Citation rendering
cite-method: biblatex
citeproc: true

# Math rendering in HTML
html-math-method:
  method: katex

from list-table.

After some investigation, I think that the problem is that the tab stop has been set to 2. Consider this example:

% pandoc <<<'  two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>

Compare with this:

% pandoc <<<'  two-indent' --tab-stop=4
<p>two-indent</p>

The CommonMark parser doesn't behave this way:

% pandoc <<<'  two-indent' --tab-stop=2 --from commonmark
<p>two-indent</p>

I don't see any suggestion in the manual that --tab-stop should affect indented code block indentation, so is this a bug in the markdown reader?

from list-table.

I made another test, but different, with both --tab-stop=2 and --tab-size=4, on Codebraid Preview and using markdown_github and other variants:

* testing... 

  :::list-table
  * - Name
    - Description
    - Origin

  * - Apple
    - Fruit
    - China

  * - Banana
    - Fruit
    - India
  :::

It worked with all the variants, except CommonMark, CommonMark X and gfm, on Codebraid Preview.

If I tab-indented the table part, it stopped working with all the variants, including CommonMark and CommonMark X, on Codebraid Preview.

from list-table.

@jgm, I'm not sure whether you saw this. Would you expect this behavior (see the last-but-one comment for more context)? Thanks.

% pandoc <<<'  two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>

from list-table.

Yes, that is expected.
John Gruber's original Markdown documentation says:

To produce a code block in Markdown, simply indent every line of the block by at least 4 spaces or 1 tab.

IF we change a tab from 4 to 2 spaces, then 2 spaces will do it.

from list-table.

Thanks. I'd expect this if there was a tab in the input, but there are two spaces, not a tab. FWIW the CommonMark parser doesn't treat this as a code block.

from list-table.

commonmark just hard-codes 4 spaces tab stop, which I think is the best plan.
--tab-stop is sort of a vestigial feature -- I doubt many people use it. What are you using it for?

from list-table.

I do not know who you are addressing to, @jgm.

As for me, to work on Codebraid Preview, I would prefer using markdown_github and any variant, except commonmark_x and gfm.

Without depending on the Pandoc Lua filter list-table, any Markdown variant works on both Codebraid Preview and the terminal.

With the dependence of the Pandoc Lua filter list-table, and with 2 or 4 tab stop spaces, commonmark_x and gfm variants do not work on Codebraid Preview, but work on the terminal. The markdown_github, markdown_mmd and markdown_phpextra variants work on Codebraid Preview and on the terminal, but only if with the 2 tab stop spaces, and with the fenced_divs and simple_tables are enabled.

However, markdown_github has more features than commonmark_x and gfm, but I reconsider switching to markdown_mmd and markdown_phpextra, but I want to stop Pandoc from wrapping a p element in every line. @mb21 recommends me to use gfm, that stops Pandoc from wrapping p element in every line, but gfm will not work on Codebraid Preview with the dependence of the Pandoc Lua filter list-table.

from list-table.

Well. I have tabe-stop in the defaults file:

preserve-tabs: true
shift-heading-level-by: -1
tab-stop: 2

Do you think that defaults file's tab-stop can be the cause of these bugs?

from list-table.

I definitely think you should try removing it, or changing it to 4!

@jgm, sorry for dragging you into this discussion. I'll create a pandoc issue to discuss the point that I raised with you.

from list-table.

Good evening, @wlupton

I have just made this test:

• With the removal of tab-stop from the defaults file or with the modification of tab-stop's value to 4, it worked on Codebraid Preview with the Markdown variant markdown_github, but not with the other variants commommark_x and gfm. I think it turns to be the bug of tab-stop.

However, I want to keep 2 tab stop spaces for the HTML elements when converting from Markdown to HTML. But it will not work with this Pandoc Lua filter on Codebraid Preview.

from list-table.

I hoped that this example from a couple of days ago would make it clear what's going on here. With a tab-stop of 2, indentation of 2 or more will be formatted as code.

% pandoc <<<'  two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>

If you want to use tab-stop=2 then you'll need to change your input file, because 3 > 2! BUT, are you sure that the tab-stop setting affects the generated HTML? I don't see why it would do!

I suggest using very simple input and defaults files (in fact, avoid using a defaults file if you can), and also making sure that it works as expected from the command-line before trying it with Codebraid.

Perhaps this example will make it clearer.

% cat spiz.md
:::list-table
   * - 3
:::

:::list-table
 * - 1
:::

% pandoc spiz.md --tab-stop=2
<div class="list-table">
<pre><code> * - 3</code></pre>
</div>
<div class="list-table">
<ul>
<li><ul>
<li>1</li>
</ul></li>
</ul>
</div>

I'm not using the list-table.lua filter here because I don't need to! It will fail, because it'll see a code block where it expects a list.

Let's assume that you have resolved the tab-stop problem, either by using tab-stop=4 (which is what I would recommend) or by editing the markdown.

Now you can try it with codebraid. You'll have (for now) to live with the restrictions explained here by @gpoore, which mean that it won't work with commonmark variants (commonmark, gfm and commonmark_x).

As I mentioned, I'd be happy to make list-table.lua work with codebraid's line numbering, but that should happen on a separate issue. I'll create this other issue tomorrow.

Does this all make sense?

from list-table.

Does this all make sense?

Yes, this does, @wlupton.

from list-table.

@gusbemacbe please could you try the feature/support-sourcepos branch with commonmark_x. Thanks.

from list-table.

Good morning, @wlupton!

I have just tested with both

• from: commonmark_x;
• from: commonmark_x-ascii_identifiers-east_asian_line_breaks-implicit_figures-pipe_tables-smart-sourcepos-wikilinks_title_after_pipe-wikilinks_title_before_pipe+attributes+autolink_bare_uris+bracketed_spans+definition_lists+emoji+fancy_lists+fenced_divs+footnotes+gfm_auto_identifiers+hard_line_breaks+implicit_header_references+raw_attribute+raw_html+rebase_relative_paths+strikeout+subscript+superscript+task_lists+tex_math_dollars+yaml_metadata_block (@gpoore's Codebraid Preview's Lua filter says not to support alerts and tex_math_gfm);

It worked perfectly and fully with Codebraid Preview with enabled and disabled scrolling, except the third table:

Only - []{rowspan=2} `italic` doesn't work because it is not spanned with two rows.

And the last table failed to work with Codebraid Preview:

:::{.list-table}
- []{style="background: lightblue"}
  - Ocean
  - Area (sq. miles)
- - Atlantic
  - 41.1 million
- - Pacific
  - 63.8 million
:::

:::{.list-table style="background: lightsalmon"}
* - Fish
  - Ocean

* :::{.list-table-body style="background: lightgreen"}
  - - Arctic char
    - Arctic
  - - Humuhumunukunukuāpuaʻa
    - Pacific
  :::

* ------------- ---------
  Cod           Atlantic
  Notothenioids Antarctic
  ------------- ---------
:::

Here is the error:

Codebraid Preview Error
pandoc failed:
Error: Command failed: pandoc --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/%40vscode/codicons/dist/codicon.css --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/vscode-markdown.css --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/codebraid-preview.css --defaults "/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/defaults/JCaZoD3OG8INzm4MPjk-TFkkqtepMQqgUZeBa0McErk.yaml" --standalone --wrap=auto --defaults "/home/gus/Tests/confluence/_codebraid/temp_defaults/_codebraid_preview.yaml" --standalone --lua-filter="/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/sourcepos_sync.lua" --katex=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/katex/dist/ --lua-filter="/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/show_raw.lua" --from "/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/readers/commonmark_x.lua-ascii_identifiers-east_asian_line_breaks-implicit_figures-pipe_tables-smart-sourcepos-wikilinks_title_after_pipe-wikilinks_title_before_pipe+attributes+autolink_bare_uris+bracketed_spans+definition_lists+emoji+fancy_lists+fenced_divs+footnotes+gfm_auto_identifiers+hard_line_breaks+implicit_header_references+raw_attribute+raw_html+rebase_relative_paths+strikeout+subscript+superscript+task_lists+tex_math_dollars+yaml_metadata_block+sourcepos" --to html
Error running filter assets/filters/lua/list-table-sourcepos.lua:
assets/filters/lua/list-table-sourcepos.lua:169: expected list item to have row attrs  at
------------------------------------------------------------------------

stack traceback:
	assets/filters/lua/list-table-sourcepos.lua:51: in upvalue 'assert_'
	assets/filters/lua/list-table-sourcepos.lua:169: in function <assets/filters/lua/list-table-sourcepos.lua:134>

from list-table.

With markdown_github and with the updated list-table filter, the spanned rows worked:

I suspend CommonMark X does not support it.

from list-table.

Thanks. The rowspan problem was my fault (I hadn't skipped the data-pos Divs in this case). It should now be fixed.

The other problem was that the table shown below is a "simple table", but commonmark_x only supports pipe tables, so I've changed this README.md example to use a pipe table.

* ------------- ---------
  Cod           Atlantic
  Notothenioids Antarctic
  ------------- ---------

I believe that all the examples are now working in Codebraid with commonmark_x. Do you agree?

from list-table.

I believe that all the examples are now working in Codebraid with commonmark_x. Do you agree?

• It almost worked, but it does not solve the colspan or rowspan problem.
• I enabled pipe_tables and it still failed:

from list-table.

Please double-check that you have the latest filter code. I believe that I have commented out the test that would lead to that error.

from list-table.

I have just tested the latest list-table codes with the latest Lua filter code on Codebraid Preview with the commonmark_x+pipe_tables and it worked fully and perfectly!

from list-table.

Great! Note that you shouldn't need the +pipe_tables (commonmark_x already enables pipe tables).

from list-table.