andymerskin / spotr Goto Github PK

It's coming! Currently working on a short example in Vue.js to demo the current functionality.

Spotr should remove stop words from the search query so they don't interfere with the field string scoring when gathering results.

Tasks

• Research common stop words
• Implement at the field stage only, since some stop words may be desired in keyword definitions

It would be cool if the filtered collection provided metadata for the search results, like scores for each result, or keywords that were matched.

Use cases:

• Create heatmaps from scores in results
• Bundle or group results based on score ranges
• Pass results to another library or plugin to be parsed or filtered further
• Display UI indicators when keywords match (or don't match)

Example:

search.query('red button up shirts with pocket')

Return value:

{
  keywords: {
    feature: {
      hasMatch: true,
      matches: ['pocket', 'button up'] // matched keyword queries
    },
    color: {
      hasMatch: false,
      matches: ['red']
    }
  },

  results: [
    {
      ...result, // result item data from collection
      score: 9.16,
      fields: {
        productName: {
          score: 0.92,
          boost: 9
        },
        productDescription: {
          score: 0.11,
          boost: 8
        }
      }
    }
  ]
}

Questions

• Is performance a concern? Maybe this data should only be surfaced when an option is passed, and by default, the results array is the only output.
  • Return value should always be an object, where the results array can be extracted in any case.

  search.query('red button up shirts with pocket', {
    metadata: true
  })

It would be awesome to have the ability to specify and pass parameters/properties to a keyword filter.

Example keyword definition for a rating filter:

keywords: {
  rating: {
    queries: [
      '{{count}} stars',
      'rated {{count}}'
      ],
    filter: (collection, {count}) => {
      return collection.filter(item => item.rating >= parseInt(count, 10))
    }    
  }
}

Questions:

• Should type conversions be handled by the user?
  • We could have it test for a parsed Number, and if true, pass as a number (feels too prescriptive).

    const numCount = parseInt(count)
    return numCount ? numCount : count

  • Users could specify types in the keyword definition. Would be useful for validation before filtering collection.

    {
      queries: ...,
      filter: ...,
      props: {
        count: Number
      }
    }

Having a Spotr instance be tied to a single collection through initial configuration, or being assigned later is an inconvenient limitation when someone wants to search similar collections using the same fields and keyword matchers. Cloning a Spotr instance or writing a duplicate configuration isn't DRY.

Use case:

• I want to search multiple collections, derived from one or more collections. For instance, I have a master collection of pages within an application, each with their own metadata. The application gives the user recently visited places or favorite places, which may be managed in separate arrays.

Example:

search.query(collection, query)
// => [filtered collection]

Error handling:

• Need to handle errors where a field doesn't exist when it's defined in fields or filters
• Need to handle empty collections

Other considerations:

• This will be API breaking, but we're in alpha, no biggy
• Remove class-level collection property. Collections must always be passed into query()
• Could memoize the query() method:

const queryRecents = query => {
  return search.query(recentsCollection, query)
}

queryRecents('balances')

Why not have Spotr generate query schemas to pass off to lodash or a Mongo-style query library like sift.js?

This would be in conflict with #6, where collections are passed into query() and return a result. Instead query would return a filter schema to pass off to a different library or routine.

OR

It may still be necessary to pass the collection into Spotr to utilize its field string matching and separating field strings from keyword queries. The resulting collection from field matching can be passed in the result payload to (sift.js, lodash). The use case would be to filter on field-matched results.

Example:

const search = new Spotr({
  fields: {
    accountName: {
      boost: 9
    }
  },

  keywords: {
    accountHoldingsAmount: {
      queries: ['accounts {{operator}} {{amount}}'],
      filter: (collection, {operator, amount}) => {
        let $operator
        
        switch(operator) {
          case 'above':
            $operator = '$gt'
            break;
          case 'below':
            $operator = '$lt'            
            break;
          case 'equal to':
            $operator = '$eq'
            break;
        }

        return {
          collection: collection,
          filter: {
            type: 'account',
            holdings: {
              [$operator]: parseInt(amount) 
            }
          }
        }
      }
    }
  }
})

search.query('custody accounts below $1000')

Returns:

{
  collection: collection,
  filter: {
    type: 'account',
    holdings: {
      $lt: 1000 
    }
  }
}

... which can be passed to sift.js, which then filters the collection it's passed.

import sift from 'sift'

const { filter, collection } = search.query('custody accounts below $1000')
const results = sift(filter, collection)

Or even better:

...
const custodyBelow1000 = sift(filter)
custodyBelow1000(collection)
// => array of custody accounts below $1000

Other considerations:

• Not to add complexity, but it may be worth exploring filter adapters, giving users the option to output Mongo queries or simply use the original keyword functions to filter on collections and return the filtered collection.
• Could be neat for generating GraphQL queries
• Does lodash have built-in memoization (to save chains or sets of operations)?

Conflicts: #6
Dependencies: #1

Currently, Spotr's field configuration requires you to pass an object with fuzzy and boost and doesn't provide defaults if you want to get up and going with Spotr immediately.

Ideally, one could configure Spotr with an array of fields, like so:

const search = new Spotr({
  collection: bigArrayOBooks,
  fields: ['title', 'genre']
})

And then queries could be scored against those fields without extra configuration.

search.query('The Great Gatsby')