I think it's a saner approach because currently I need to maintain a fork of the default UI at: https://gitlab.com/g.grossetie/antora-ui-default/-/jobs/artifacts/lunr-integration/raw/build/ui-bundle.zip?job=bundle-dev

Original idea by @eskwayrd in https://github.com/eskwayrd/antora-indexer-poc

Otherwise the supplemental_ui files must be copied from GitHub.
We should also update the documentation.

Hello,

Thanks for this work!

After I followed the instructions in the README and started to customise the default site generator, a Google search revealed the NPM site generator to me.

This should be mentioned as one, if not the preferred way to test the project.
The setup of some of the least techy users could be messed up with the local customisation.

@Mogztter Thanks, I followed the README and got lunr integrated on my Antora 2.0 development setup. There were a few places where I found that the instructions could be clearer (or be updated).

https://github.com/Mogztter/antora-lunr/blob/ead8207ac0e441848ae032c02006b32f240e7c8a/README.adoc#L56

• This is now site/search_index.json.

https://github.com/Mogztter/antora-lunr/blob/ead8207ac0e441848ae032c02006b32f240e7c8a/README.adoc#L62-L68

• I got the impression that I should edit the url, instead of just replacing it with the one you gave.
  • Meaning, edit it to work from my fork!
• Something like "Use the following URL..." would be clearer.

https://github.com/Mogztter/antora-lunr/blob/ead8207ac0e441848ae032c02006b32f240e7c8a/README.adoc#L70

• I couldn't make any progress when attempting to use a file:// reference.
• However, using serve and setting url: http://localhost:5000 in site.yml worked beautifully.
• From https://docs.antora.org/antora/2.0/playbook/configure-site/#configure-url:

  • The base URL should not end with a trailing slash.

Thanks for sharing this awesome tool!

Is there any docker image based on your antora-site-generator-lunr?
That would run exactly as the antora/antora image?

PS: I have only found this thread on gitter.im regarding the topic.

Hi this is not an issue. Apologies for that.

I got this working locally but I have no idea how to integrate on gitlab.

Could you please have a look and help me?

Repo
Playbook
Pipeline

The indexer should skip pages which are marked as "noindex". For example, consider a page which has the following meta tag:

<meta name="robots" content="noindex">

Another option is to honor exclusions in robot.txt (or perhaps it should be both).

Here's what Google has to say about noindex: https://support.google.com/webmasters/answer/93710?hl=en

Here are some other ideas around inclusion/exclusion to consider: https://swiftype.com/documentation/site-search/crawler-configuration/content-inclusion-exclusion

Hello,

I would like to remove h6 titles from lunr generated search-index. Is that in any way possible? I've tweaked the generate-index.js and removed the h6 entry from the following line (line 64):
$('h2,h3,h4,h5', article).each(function () { but that doesn't seem to work.

Any suggestions.

Hi,
thank you for easy setup and use of this project.
Would you have an idea how to get a improved search results overview in case multiple versions of the documentation is being built? Would there be an option to group search results based on the version?

Thank you.

Should you consider only indexing content in the .doc class?

Take a look at what I've done here: campbellbartlett@72526c8

This prevents things like the header, footer and menu from being indexed, which is present on every page.

Hello,

we are interested in what is new in the master branch, especially the fact that it is possible to have an empty URL in the playbook, and that it will then create relative URLs: https://github.com/Mogztter/antora-lunr/blob/master/lib/generate-index.js#L23

Indeed, we find it to be a very interesting purpose to have a website that can be hosted at several URLs (for testing, for UAT, for prod), without having to create separate bundles for each environment.

Do you think the project is in a stable enough state that a new version could be produced and released to npm?

If the site URL is set, only prepend the pathname segment (i.e., context path) to the entries instead of the full value. For example, if the site.url value is https://example.org/docs, only use /docs. If the site URL does not have a pathname, the entries should begin with / (but not //).

Only using the pathname makes the index portable as the site is promoted across parallel environments (testing, acceptance, production) or hosted via multiple domains (to achieve high availability).

In brief, the domain segment doesn't matter. What matters (for the browser to assemble the correct URL) is the pathname that comes after the domain.

Using the pathname where possible is now standard practice in Antora. Antora will only use the full value of the site.url if an absolute URL is required (such as for the canonical URLs or in sitemap entries).

As of now, antora-lunr does not support partial word search. Lunr supports using wildcards, so implemntation could be as simple as adding text = "*" + text + "*" in function searchIndex.

The Lunr index can quickly weight a few megabytes. Since the client will have to download this file, we need to find solution to reduce the size of this file.

Using compression https://github.com/nodeca/pako ?

Current version is 2.3.3, the latest is 2.3.8 which offers some bugfixes around fuzzy matching. Happy to put together a PR for this...

Perhaps I'm fooling myself, but it seems really easy to make the search completely portable/relocatable. Commit 94185cb in my plugin-377 branch has the changes, but there's also a bunch of lint fixes to search.js

The crucial lines are, in generate-index.js: near line 99

         component: page.src.component,
         version: page.src.version,
         name: page.src.stem,
-        url: urlPath + page.pub.url,
+        url: page.pub.url,
         titles: titles // TODO get title id to be able to use fragment identifier
       }

and in search.js: near line 112

     var documentHit = document.createElement('div')
     documentHit.classList.add('search-result-document-hit')
     var documentHitLink = document.createElement('a')
-    documentHitLink.href = item.ref
+    var rootPath = window.antora.basePath
+    documentHitLink.href = rootPath + item.ref
     documentHit.appendChild(documentHitLink)
     hits.forEach(function (hit) {
       documentHitLink.appendChild(hit)

Is there some reason to use absolute urls in the search?

Generated search-index.js always contains only relative paths and ignores site.url setting in the playbook.

Currently, this library is available as a custom site generator (through https://github.com/Mogztter/antora-site-generator-lunr).

With the introduction of the pipeline extension feature in Antora 3, we can simplify the integration.
The pipeline extension provides a lightweight event-based system that we can leverage to generate a search index.

I think we should register our extension on beforePublish:

module.exports.register = (pipeline, { config }) => {
  pipeline
    .on('beforePublish', ({ playbook, contentCatalog }) => {
      // generate a search index
    })
}

Additionally, extensions can be configured: https://docs.antora.org/antora/3.0/extend/pipeline/configure-extension/ so we could potentially address the following issues:

• #33
• #75

I had some problems following the instructions, specifically:
1.antora-lunr should be installed globally if antora is also installed globally (npm install -g antora-lunr)
2. Extend the bottom note by saying the search will not work due to the CORS issue if you view your site locally without having a http server.

Thanks for the work so far!

• https://docs.fauna.com/
• https://documentation.suse.com/external-tree/en-us/suma/4.0
• https://www.uyuni-project.org/uyuni-docs/

Feel free to add a link to your documentation site.

Hello @Mogztter

Thanks for your work on antora-lunr and other projects I used.

I quickly integrate antora-lunr through antora-site-generator-lunr on my website. https://jabby-techs.fr

As you can see, my website is in french and by default lunr only support english. There are some language plugins available to support french and other languages. See https://lunrjs.com/guides/language_support.html

I would like to know if there is a way to add by some configuration those languages into antora-lunr. The only way I found is to create my own generate-site function but I think it's better to improve this project.

Ewan did a great job in https://github.com/eskwayrd/antora-indexer-poc and I think we should merge the two projects.

@eskwayrd What do you think ?

I was unable to build my fork of the Antora default UI after merging in lunr support until I fixed 480 eslint errors in js/vendor/lunr.js, 3 in js/vendor/search.js, and 3 css lint errors in css/search.css.

Is it possible that the generated search URLs are ignoring the Antora site.yml file's site.url setting? I'm getting only relative URLs generated no matter what I have site.url set to.

Thanks for this great project, BTW, and for the recent update.

I would like to be able to index only the latest component version (but not the master/unstable branch) for two reasons:

• Reduce index size. Many versions which are similar will quickly generate a large index (I'm not sure how well this is compressed) and I'd like to see what difference it makes if I index a single version.
• Improve UI display. The present results display with antora-lunr is a bit repetitive and just the latest version is an easy workaround for a difficult UI problem (I guess results on the same page across multiple versions should be squashed into a version tag which then itself becomes repetitive).

I think the solution is to iterate over pages and find the latest version and only index that.

As I understand the code (which is not very well) at this point in the generation, the virtual files are already in html and so either we need a reverse lookup to extract the version for the html or we need to find this information within the web page. I thought there might be a data- attribute or a meta attribute with this information but it's not obvious to me that this currently exists.

It may also be useful to extract the prerelease information for some purposes.

The search functionality stopped working for me since 0.7.1. It looks like the site.url which I set in the antora-playbook.yml isn't passed to the search module. Here is my test: Antora Demo Site. I built the site with DOCSEARCH_ENABLED=true DOCSEARCH_ENGINE=lunr DOCSEARCH_INDEX_VERSION=latest NODE_PATH="$(npm -g root)" antora --generator antora-site-generator-lunr antora-playbook.yml. When I search for a word and click on one of the variants it gives me a URL like this: file:///component-b/2.0/module-one/overview.html -- i.e. without the site.url part which I set in the playbook to file:///Users/pavela/tmp/docs-site/build/site.

I'm running the latest versions of everything:

$ npm ls -g --depth 0
/Users/pavela/.nvm/versions/node/v12.17.0/lib
├── @antora/[email protected]
├── @antora/[email protected]
├── [email protected]
├── [email protected]
└── [email protected]

Last time it worked properly was about a month ago (I didn't test since then). My suspicion is that it might be caused by this commit: 364a957

Appreciate your help.

The following document contains two first level headings: https://gitlab.com/antora/antora/edit/master/docs/modules/ROOT/pages/whats-new.adoc

Currently we concatenate the two titles (without a separator):

For reference, Algolia uses the first title:

Does Lunr support Chinese search? If not, how can I make it support?

I've configured lunr as described in the readme and it doesn't work propperly (I'm running it in Windows).
Lunr should generate searc-index.js, but all I get is search_index.json.gz.
Am I doing something wrong?

I followed the documentation here (using the antora-site-generator-lunr) and manage to have a working website with the search bar! Very well explained! Kudos!

However, all the pages were containing duplicated ToC side panel!

In order to fixed it, I only have to remove the following line from footer-scripts.hbs:

<script src="{{uiRootPath}}/js/site.js"></script>

The browser debugging mode shows that the site.js is still loaded.. probably because of a newer ui-bundle.zip from antora-ui-default?

For anyone building on Windows like me, can we add a sidebar or the like noting the following:

Terminal / Command Prompt / PowerShell

To use PowerShell, the user needs to change the execution policy (as an Administrator) with

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine

Alternatively, use cmd.exe.

Setting Environment Variables

Environment variables must be set apart from the antora call. As so:

C:\docs>set DOCSEARCH_ENABLED=true
C:\docs>set DOCSEARCH_ENGINE=lunr
C:\docs>antora antora-playbook.yml

An asciidoc-related project using markdown??

Hi! Is there a way to correlate the titles inside a document with the metadata in the search result?

UPDATE: I just saw https://github.com/Mogztter/antora-lunr/blob/13e6afea4f63b9dad88ff4549f8d3403f15f24a5/lib/generate-index.js#L83, can you give a quick explanation of what you have in mind?

After following the instructions in README to build the official Antora doc the site could be generated successfully without errors and the search-index.js file was generated successfully in the root directory, but I didn't see the search box in the UI. I don't understand this line in README "Copy the supplemental_ui directory in your Antora playbook repository and configure a supplemental_files". The supplemental_ui directory is already there. What place should I copy it to?