Learn Full Stack
with tapaScript
By clicking Sign Up you re confirming that you agree with our Terms and Conditions.
A small chunk of text a model processes — a word or part of a word.
A small piece of text that an AI model processes; it may be a word, part of a word or punctuation.
💡 Cutting a sentence into small Lego blocks before assembling and understanding it.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("What is a token?")
print(len(tokens)) # e.g. 5 tokensToken usage affects API costs, context limits and processing time.
Tokens are not words — a single word can be split into several tokens.
with tapaScript
By clicking Sign Up you re confirming that you agree with our Terms and Conditions.
