Make markovify.Text accept a file-like object or generator to reduce memory footprint when using large files? about markovify HOT 3 CLOSED

Thanks for suggesting this. It's an improvement I'd been meaning to make. Now available in v0.6.1. Fetch the latest with pip install -U markovify, and see the instructions here: https://github.com/jsvine/markovify#generating-markovifytext-models-from-very-large-corpora

Does that improve performance for you?

from markovify.

I'm just unsure how to deal with sentence splitting along chunks.

I'd recommend using a generator internally for this, where it runs over an iterable (a generator, list, or something else) and only yields a new sentence upon the discovery of a !, ? or ..

That way you're relying on Python for maintaining state for you, rather than maintaining state with local variables.

from markovify.

Also you may try PyPy for some speedups

from markovify.