Formalise the backtracking helper method in lexer

- Allow the specification of number of tokens, and document how that is
  used inside the relevant test case.