I liked this tutorial, thank you. Two suggestions that I implemented in my own version:

1. Right now there is no precedence among lexing rules, because you store them in a dictionary and the first one that matches (if any) wins. In practice, it’s possible for more than one lexing rule to match. The heuristic that the Racket lexer uses (for example) is that the longest match takes precedence, and if there are multiple matches with the same length, the earlier rule takes precedence.

2. I don’t think you need trimLeadingWhitespace. You can just make a .whitespace token that matches \\s+. On the right side of that lexing rule, return nil. In other words, you tokenize the whitespace and then throw it away.

Thanks again!