This repository contains a simple wrapper to use Google's Universal Sentence Encoder with Runway.
The Runway model exposes one command, embed
, that
accepts a text
parameter as input and produces two arrays as output: one with
an array of sentences in text
, and another with sentence embeddings for each
sentence with the corresponding index in the array of sentences.
An optional tokenize_sentences
parameter, if set to yes
, uses NLTK's
sent_tokenize
function to parse the value in text
into sentences before
passing them along to the embedder. Otherwise, text
is interpreted as a list
of sentences separated by newline characters.
This code uses the Cross-lingual (XLING) Universal Sentence
Encoder,
which Google advertises as working with English, French, German, Spanish,
Italian, Chinese, Korean, and Japanese. But note that the sentence tokenizer
used in this port is specific to English, and will not return good results for
other languages! If you're getting embeddings for sentences from other
languages, you may want to set the tokenize_sentences
parameter to no
and
feed the model pre-tokenized sentences.