add dynamic token buffer
- TerenceP
Owned by TerenceP
Add a token buffer that does not consume all of the input upfront so that we may handle arbitrarily long strings.
Add a token buffer that does not consume all of the input upfront so that we may handle arbitrarily long strings.