How to only return what can be parsed in a TokenizerBuilder about superpower HOT 2 CLOSED

Hi @hwalkenshaw,

There's no direct way to do it, but you might have some success declaring a SqlToken.Unknown value.

First, though - the exception you're hitting is because Character.Letter.Many() allows zero-width matches, i.e. "" would be an infinite set of SqlToken.Keywords using this matcher. Superpower picks that up and throws an exception in that case.

You probably want Character.Letter.AtLeastOnce() rather than Many() in this rule.

To add SqlToken.Unknown looks something like:

            var tokenizer = new TokenizerBuilder<SqlToken>()
                .Ignore(Span.WhiteSpace)
                .Match(Character.Letter.AtLeastOnce(), SqlToken.Keyword)
                .Match(Character.Matching(ch => !char.IsWhiteSpace(ch), "token content"), SqlToken.Unknown)
                .Build();

HTH!

from superpower.

Howdy! I'll assume the above got you unblocked; let me know if it raised more questions than it answered :-) .. Cheers!

from superpower.