ability to load synonyms from files (like stop word lists)
ability to either expand (index all synonyms)
or contract (consolidate synonyms to single version)

also, investigate wordnet: http://wordnet.princeton.edu/

truncate token at the specified max lenght
useful for fields left as a single token

will improve analyzer for french

While we can't use them for the index keys which we craft to get the desired sort order, we should use protobufs to encode the index values. This will make the binary serialization/deserialization less error prone, more compact, and easier to evolve over time.

initial wiki pages for:

• building bleve with all the c libraries
• creating a custom mapping
• one for each type of field
• one for each type of analyzer/character filter/tokenizer/token filter
  • these can be stubs initially, but serve as place holders for adding information over time
• one for each type of query

arabic, german, hindi, indic, kurdish, persian, and scandanavian

Currently the back index contains 2 separate lists of more strongly typed data. This should be changes to just a flat list of keys. This will make it easier to introduce new index row types in the future without having to keep updating the way the back index works.

will improve turkish analyzer

use unicode/utf8 package RuneCount method
also rename existing length filter to ByteLength

Initial implementation should just operate at query time.

If we swap the field and term order in the index key we can support faceting at query time. For every document satisfying the original query, we can look up the document in the back index, and find entries for the field that is being faceted. Seems like we don't even have to load that key, just be able to parse the field id and terms. For categorical facets the terms are bucketed and counted. For numerical range facets the parsed terms are bucketed and counted. The top-N facets are then returned with the query results.

Currently index term entries are:

't'

Would like to add support for also storing the position of this term in any arrays that were a part of the field path.

Not 100% decided that this must be in the key, but that would be the only way to have some hope of efficiently querying on this information.

The idea is to be able to further qualify queries and say that in addition to other query criteria, matching items must occur in the same parent element.

Consider the following documents in an index.

{
  "name": "a",
  "children": [
      {
          "name": "c",
          "age": 25
     },
      {
          "name": "d",
          "age": 15
     },
}

{
  "name": "b",
  "children": [
      {
          "name": "c",
          "age": 15
     },
      {
          "name": "d",
          "age": 25
     },
}

Logically we want to query:
child.name = "c" AND child.age < 20 AND same child

Both documents have a child named "c" and a child who's age is less than 25, but ONLY "b" satisfies both criteria in the same child.

The implementation idea is to include the position in the children array, and the query criteria "same child" is accomplished by verifying that matching items have the same value.

options min length, max length
for each input token, compute ngram tokens based on parameters, emit all resulting tokens

The top-level bleve package should be all one needs to import to achieve the following:

• create new index
• open existing index
• create new default mapping
• modify default mapping into custom mapping
• index document/object

• fold fullwidth ASCII variants into the equivalent basic Latin
• fold halfwidth Katakana variants into the equivalent Kana

See also full ICU analysis

Users should be able to query an index using nothing but the top-level bleve package

Need to ensure we have the ability to store things like ngram entries and not confuse them with indexed terms.

Depends on #19

update readme to reference google group

depends on #19 and #20

ability to tag words as keywords
keywords should then be ignored by the stemmer

options min length, max length, side (front/back)
for each input token, compute ngram tokens based on parameters, emit all resulting tokens

Once the new API is done, the project README should illustrate how easy it is to index data and query it.

See http://www.elasticsearch.org/blog/stop-stopping-stop-words-a-look-at-common-terms-query/

Two modes:

1. Return terms which start with this prefix
2. Return documents which contain a term starting with this prefix

Depends #23