Supporting Jekyll highlighting about liqp HOT 18 CLOSED

Nope, it is not present in Liquid, only in Jekyll. I'll have a look at it later on.

from liqp.

Wait, you can simply add the tag at runtime:

Template template = Template
        .parse("{% highlight ruby %} puts 'something' {% endhighlight %}")
        .with(new Tag("highlight") {

    @Override
    public Object render(TemplateContext context, LNode... nodes) {

        String language = nodes[0].toString();
        String source = super.asString(nodes[1].render(context));

        return "TODO: display `" + source + "` as " + language;
    }
});

System.out.println(template.render());

// prints: TODO: display ` puts 'something' ` as ruby

from liqp.

Cool... should I do that or will you add this as part of liqp as a flavor support or so?

from liqp.

I thought about adding it, but that would involve more: I'd need to provide a mechanism/interface to hook into the render(...) method of this built-in highlight tag since integrating the actual highlighting library (or libraries) would be too much. So, I recommend doing it as I did above.

from liqp.

So if I understand you correctly the best approach for me would be to create a new Template class e.g. called JekyllTemplate that overrides render like above. But would it not also have to call the super render method? Actually.. it would have to add the new tag and I would have to implement the tag.. correct?

from liqp.

I am trying this now... will also have to add support for linenumbers..

from liqp.

No, there is no need to subclass Template. You just process your jekyll (or liquid) source files as normal, like this:

String rendered = Template
    .parse("{% highlight ruby %} puts 'something' {% endhighlight %}")
    .render();

but then do this:

String rendered = Template
    .parse("{% highlight ruby %} puts 'something' {% endhighlight %}")
    .with(new Highlight())
    .render();

where Highlight looks like this:

public class Highlight extends Tag {

    public Highlight() {
        super("highlight");
    }

    @Override
    public Object render(TemplateContext context, LNode... nodes) {
        
        String language = nodes[0].toString();
        String source = super.asString(nodes[1].render(context));
        
        // your logic here to syntax-highlight `source` (if language equals "ruby")
        
        ...
    }
}

from liqp.

Actually.. it would have to add the new tag and I would have to implement the tag.. correct?

Yes, that is it.

from liqp.

So I have a class now that does this to some degree:

package io.takari.idiom.jekyll.liquid;

import liqp.TemplateContext;
import liqp.nodes.LNode;
import liqp.tags.Tag;

/**
 * Highlight tag for Jekyll source.
 * 
 * - does not support language specific tagging within code snippet
 * - does not support linenos argument
 * - currently collapses multi-line code into one (still have to see if this is css or code related)
 * 
 * @see https://jekyllrb.com/docs/posts/#highlighting-code-snippets
 * @see https://github.com/bkiers/Liqp/issues/42
 * @author Manfred Moser - [email protected]
 */
public class Highlight extends Tag {

  public Highlight() {
    super("highlight");
  }
  @Override
  public Object render(TemplateContext context, LNode... nodes) {
    String language = nodes[0].toString();
    // add support for linenos later
    String source = super.asString(nodes[1].render(context));

    StringBuilder builder = new StringBuilder()
      .append("<figure class=\"highlight\">\n")
      .append("<pre>\n")
      .append("<code class=\"language-" + language + "\" data-lang=\"" + language + "\">")
      .append( source )
      .append("</code>")
      .append("</pre>")
      .append("</figure>");
    return builder.toString();
  }
}

but there are two problems I am struggling with.

1. nodes[1].render(context) looses the newline from the code snippet so the original code snippet is collapges to a one-liner.

E.g. try with something like

{% highlight shell %}
cd /opt/dev/myproject
mvn clean install
{% endhighlight %}

2. if my highlight block also has a linenos argument it bails out and never even reaches my tag impl.

{% highlight shell linenos %}
cd /opt/dev/myproject
mvn clean install
{% endhighlight %}

instead I get

Exception in thread "main" java.lang.RuntimeException: UnwantedTokenException(found=linenos, expected 91)
at liqp.parser.LiquidParser.reportError(LiquidParser.java:171)
at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:603)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at liqp.parser.LiquidParser.custom_tag(LiquidParser.java:948)
at liqp.parser.LiquidParser.tag(LiquidParser.java:655)
at liqp.parser.LiquidParser.atom(LiquidParser.java:439)
at liqp.parser.LiquidParser.block(LiquidParser.java:296)
at liqp.parser.LiquidParser.parse(LiquidParser.java:202)
at liqp.Template.(Template.java:80)
at liqp.Template.parse(Template.java:153)

from liqp.

nodes[1].render(context) looses the newline from the code snippet so the original code snippet is collapges to a one-liner.

Hmm, when I run the following class:

import liqp.nodes.LNode;
import liqp.tags.Tag;

public class Test {

    public static void main(String[] args) {

        String source = "{% highlight shell %}\n" +
                "cd /opt/dev/myproject\n" +
                "mvn clean install\n" +
                "{% endhighlight %}";

        String rendered = Template
                .parse(source)
                .with(new Highlight())
                .render();

        System.out.println(rendered);
    }
}

class Highlight extends Tag {

    public Highlight() {
        super("highlight");
    }

    @Override
    public Object render(TemplateContext context, LNode... nodes) {
        return super.asString(nodes[1].render(context));
    }
}

I get the following output:

cd /opt/dev/myproject
mvn clean install

Perhaps you're looking at the HTML? You'll need to replace the \n with <br>, I guess.

if my highlight block also has a linenos argument it bails out and never even reaches my tag impl.

Looking at the grammar for a custom tag: TagStart Id (expr (Comma expr)*)? TagEnd, it looks like there needs to be a comma between shell and linenos: {% highlight shell, linenos %} .... I don't know if that is Liquid syntax or a bug (the mandatory comma). I'll have to look at that. For now, it's past my bed time, so says my girlfriend :)

from liqp.

Thanks for all the great help @bkiers ! I will check that out.

from liqp.

So if there is a comma between shell and linenos it works. I adpated my code for that. The newlines from the markdown source I am parsing are gone .. somehow they are getting lost in earlier parsing so thats my own problem at this stage. If you could figure out whats up with the required comma that would help. Also let me know if you want a separate bug for that.. and I will share my Highlight class once its a bit more solid..

from liqp.

Turns out that adding a comma before linenos is invalid in Jekyll ... is that a Jekyll bug or a Liqp bug?

I get

Liquid Exception: Syntax Error in tag 'highlight' while parsing the following markup: shell, linenos Valid syntax: highlight [linenos] in docs/getting-started/markdown-guidelines.md
/var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/tags/highlight.rb:19:in `initialize': Syntax Error in tag 'highlight' while parsing the following markup: (SyntaxError)

shell, linenos

Valid syntax: highlight [linenos]
from /usr/lib/ruby/vendor_ruby/liquid/tag.rb:9:in new' from /usr/lib/ruby/vendor_ruby/liquid/tag.rb:9:in parse'
from /usr/lib/ruby/vendor_ruby/liquid/block.rb:31:in parse' from /usr/lib/ruby/vendor_ruby/liquid/tag.rb:10:in parse'
from /usr/lib/ruby/vendor_ruby/liquid/document.rb:5:in parse' from /usr/lib/ruby/vendor_ruby/liquid/template.rb:122:in parse'
from /usr/lib/ruby/vendor_ruby/liquid/template.rb:108:in parse' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/liquid_renderer/file.rb:11:in block in parse'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/liquid_renderer/file.rb:47:in measure_time' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/liquid_renderer/file.rb:10:in parse'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/renderer.rb:129:in render_liquid' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/renderer.rb:82:in run'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/site.rb:463:in block in render_pages' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/site.rb:461:in each'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/site.rb:461:in render_pages' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/site.rb:191:in render'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/site.rb:69:in process' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/command.rb:26:in process_site'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/commands/build.rb:63:in build' from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/commands/build.rb:34:in process'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/lib/jekyll/commands/serve.rb:37:in block (2 levels) in init_with_program' from /var/lib/gems/2.3.0/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in block in execute'
from /var/lib/gems/2.3.0/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in each' from /var/lib/gems/2.3.0/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in execute'
from /var/lib/gems/2.3.0/gems/mercenary-0.3.6/lib/mercenary/program.rb:42:in go' from /var/lib/gems/2.3.0/gems/mercenary-0.3.6/lib/mercenary.rb:19:in program'
from /var/lib/gems/2.3.0/gems/jekyll-3.3.1/exe/jekyll:13:in <top (required)>' from /usr/local/bin/jekyll:23:in load'
from /usr/local/bin/jekyll:23:in `

from liqp.

I'll have a look at that comma tomorrow.

from liqp.

Sounds great .. but no rush .. I am off for a few days..

from liqp.

Looking at the Liquid docs:

Create your own tags

[...]

The second argument is whatever arguments were passed to your tag, as a single string.

• Note that Liquid provides no standard argument handling for custom tags (or even for its own built-in tags) — all you get is unstructured text, so you're in charge of determining how many arguments you got and validating their values.

• Liquid also provides no standard way to indicate whether an argument is meant to be an expression or a literal value, and it won't replace expressions with their value before passing them into your tag. Thus, if you need to treat an argument as an expression, you'll have to do something hairy like test it against the Liquid::VariableSignature constant and then call Liquid::Variable.new().render(context) somewhere in your own render method.

https://github.com/Shopify/liquid/wiki/Liquid-for-Programmers

It appears that the mandatory comma is a bug in my grammar. As of now, the grammar expects the parameters of custom tags to be expressions, whereas the Liquid docs state that it should just be a plain (single!) string. This needs a bit of refactoring of the grammar and tree-walker on my end.

In your case, that would mean that ruby linenos in {% highlight ruby linenos %} puts 'something' {% endhighlight %} would be passed as a single string.

from liqp.

Fixed in the PR referenced above ☝️

from liqp.

Great - thx

from liqp.