In the docs you only index one record, but a search engine won't be very useful unless you can index a lot of them.

What would you think of adding an example of how to index an array of records?

1. I forgot to expose them ElmTextSearch module, they are only in Index at moment.
2. I have to figure out how to expose the errors types with constructors correctly.

The text readout at the end of the MultipleAddSearch.elm example should be:
"Result of searching for "title" is " ++ SearchResults

The example searches for the keyword "title" not "explanations".

If you run the example in the readme it finds a result, but if you then remove the last two characters from "explanations" so you have "explanatio" then it won't find anything. This is different from lunr.js behavior.

The API docs seems to suggest that this should not happen:
Each token is expanded, so that the term "he" might be expanded to "hello" and "help" if those terms were already included in the document index.

http://package.elm-lang.org/packages/rluiten/elm-text-search/2.0.0/ElmTextSearch#search

Is this a bug or am I misunderstanding the docs? :)

I'm indexing a FAQ archive.

In my prototype I have "Question1" and "Question2" and when I search for "Q" I get nothing, but if I rename "Question2" to "Puestion2", then I find "Question1". This seems like a bug.

I have a use case where it would be very useful if the index was able to use fields that hold lists of strings in addition to single strings and apply the same weight to all strings in the list. The documents in this use case look like:

{ cid : String
, name : String
, synonyms : List String
}

So I'd like to index both the name and the synonyms and apply the same weight to all synonyms.

From a SimpleConfig / Config API, I would see it as adding a new option, something like:

list_fields : List (doc -> List String, Float)

Which would be used like this:

ElmTextSearch.new
{ ref = .cid
, fields = [ ( .name, 5.0 ) ]
, list_fields = [ ( .synonyms, 1.0 ) ]
}

• Index.elm
• IndexDefaults.elm
• IndexLoad.elm
• IndexModel.elm
• IndexUtils.elm
• IndexVector.elm

should be

• Index.elm
• Index/Defaults.elm
• Index/Load.elm
• Index/Model.elm
• Index/Utils.elm
• Index/Vector.elm

I've encoded an Index to JSON, and then decoded it, which produces a CodecIndexRecord. How do I use that to get back an Index?

I am indexing documents with this type:

type alias RecipeName = String 

Type alias RecipeBody = String 

type alias FeuilleLiaison =
    { date : Date
    , filename : String
    , wholeBody : String
    , recipes : Dict RecipeName RecipeBody
    }

I am using two indexes, one for whole text search, one for the recipes only.
As each document can contain zero, one or more recipes I thought of setting up the index like this:

recipeConfig =
    { indexType = "ElmTextSearch - Customized French Stop Words"
    , ref = Date.toIsoString << .date
    , fields =
        []
    , listFields = [ ( Dict.keys << .recipes, 5.0 ), ( Dict.values << .recipes, 1.0 ) ]
    , initialTransformFactories = Index.Defaults.defaultInitialTransformFactories
    , transformFactories = [ (\func index -> ( index, func )) (FrenchStemmer.stemmer True) ]
    , filterFactories = [ createFilterFunc frenchStopWords ]
    }

The search is working fine and both recipes names and bodies seem to be indexed. However the scores associated with the results are all equal to NaN, so I cannot sort or filter the results.

Putting ( always "", 1 ) in fields seems to remove the NaNs but I do not know if it affect the search in any way.

Am I using the listFields parameter wrong?

I have a use case where I receive documents as part of a HTTP GET that may or may not already be in my index so I'd like to add them if they're not in and update if they're already in. The way I do this at the moment is to add and if this results in an error, update. It would be nice to have a addOrUpdate capability instead.

Wondering if you can do wild cards like llo or *ll

Also, can you use AND | OR

p.s. Thanks for a great library!

When a search is done for the single letter "a" it produces the error: "Error after tokenisation there are no terms to search for."

This can be reproduced by changing the search term in the MultipleAddSearch.elm example to either "a" or "A".

"A" was the only character I found that caused a problem. Other single letters or numbers work correctly.

I am using the library to index a corpus of recipes in French. I replaced the default stop word list by a French one, so far so good.

In French we often use l' or d' as determiner before nouns as in de l'eau, l'ail meaning some water and the garlic. I think the tokenizer includes l' or d' with the following words, so a search for ail or eau does not work, probably because the search string tokens are expanded to the right. The fact that searching for john in a text containing John's book works seems to confirm that.

Ideally the index would only register ail and eau as tokens and ignore what comes before, as it it not relevant in the context of a search.

Is there a way to change the index config in order to achieve this effect?

In my initial testing, I got an error when I searched for the text "one". As far as I can tell, it looks like processTokens transforms the tokens before filtering them. In this case, "one" becomes "on" and is filtered out as a stopword.

I don't know much about this stuff, but it would seem to make sense to filter out stopwords first? This seems to be what lunr.js does according to this comment

Example: https://ellie-app.com/346jy8VmsMXa1/0

Hey @rluiten! Are you planning to continue maintaining this and related packages into Elm 0.19? A response either way will help us at NoRedInk to plan our codebase upgrade. Thank you!

Is there a recommended method to retrieve the actual documents after you get the results? I always struggled with this with lunr.js as well...to me, when I do a search, the IDs aren't easily mapped to the documents they represent since the original documents are in a list rather than a Dict.

Obviously you can do a map over the results then filter the original list to pull out each document, but this seems inefficient.

Great port btw! Working great, just though I'd post up with this question!

I'm trying to handle some of the errors in certain ways, but having to switch on strings is a bit of a bummer. Would you be interested in a PR that replaces the string error messages with some custom union types?

I'm picturing something verbose but straightforward like these (we could sort out the details to your taste):

type AddError
    = UniqueRefIsEmpty
    | NoTermsToIndexAfterTokenisation
    | DocAlreadyExists

type RemoveError
    = UniqueRefIsEmpty
    | DocIsNotInIndex

type SearchError
    = IndexIsEmpty
    | QueryIsEmpty
    | NoTermsToSearchAfterTokenisation

I went to https://elm-lang.org/try and added rluiten/elm-text-search as a dependency.

This snippet demonstrates the issues by performing 3 searches and printing out the number of results for each lo, loy, and loya respectively. You can see that only for loy, I'm getting no results back.

import ElmTextSearch
import Html

main =
  let
    index =
      ElmTextSearch.new
        { ref = .id
        , fields = [ (.title, 1 ) ]
        , listFields = []
        }

    indexAddResult =
      ElmTextSearch.add { id = "1234", title = "Loyalty" } index
      
    searchResultLo =
        indexAddResult
          |> Result.andThen (\i ->
            ElmTextSearch.search "lo" i |> Result.map (Tuple.second >> List.map Tuple.first)
          )
          |> Result.map List.length
          |> Result.withDefault 0  
      
    searchResultLoy =
        indexAddResult
          |> Result.andThen (\i ->
            ElmTextSearch.search "loy" i |> Result.map (Tuple.second >> List.map Tuple.first)
          )
          |> Result.map List.length
          |> Result.withDefault 0
          
    searchResultLoya =
        indexAddResult
          |> Result.andThen (\i ->
            ElmTextSearch.search "loya" i |> Result.map (Tuple.second >> List.map Tuple.first)
          )
          |> Result.map List.length
          |> Result.withDefault 0          
  in
  Html.ul []
    [ Html.li [] [Html.text (String.fromInt searchResultLo)]
    , Html.li [] [Html.text (String.fromInt searchResultLoy)]
    , Html.li [] [Html.text (String.fromInt searchResultLoya)]
    ]