This is a Java implementation of a GPT3/4 tokenizer, loosely ported from Tiktoken with the help of ChatGPT. ...that all 3.5-turbo ...

Since billing is based on tokens, it would be very helpful to be able to measure how many input and output tokens are used by a ...