State Size Limit about markovify HOT 3 CLOSED

Good question! There's not a hard-coded limit, but you'll definitely run into problems with large state limits. Small-ish corpora won't have many sequences that share state_size - 1 consecutive items — e.g., it's much easier to find a subsequent item for John ate than for John ate a bagel with.

Some convenience methods, e.g. markovify.Text.make_sentence(...), handle this issue by specifying a maximum number of times to try creating a new chain. That said, it's definitely possible that there's some hidden bug causing problems.

Do you have example code/data to reproduce the problem?

from markovify.

Here is the block of code.

file_path = 'resources/the-elves.txt'
with open(file_path) as f:
        file_content = f.read()
model = markovify.Text(file_content, state_size=6)
return ' '.join([model.make_short_sentence(150) for i in range(5)])

This runs fine with a state size of 5 or less. From your explanation above, I understand why a large state size can be an issue.

from markovify.

Thanks! Do you have a link to the "resources/the-elves.txt" file? If so, I can try to figure out what's causing this particular problem. (If there's no link, you can also email me the file: [email protected].)

from markovify.