add dynamic token buffer

Add a token buffer that does not consume all of the input upfront so that we may handle arbitrarily long strings.