Token
The basic unit of text that an AI model processes, which can be a word or part of a word.
Detailed Definition
A token is the basic unit of text that an AI model understands. For LLMs, this is sometimes a word, but often just a part of a word. For example, "ChatGPT is smart." might be split into the tokens "Chat," "GPT," "is," "smart," and "." Even though "ChatGPT" is one word, the model breaks it into smaller pieces to make learning language more scalable, flexible, and efficient. There are analogous concepts like "patches" for image models and "frames" for voice models.
Core ConceptsMore in this Category
Action Model
An internal model used by AI agents to predict the potential outcomes of their actions in specific states.
Agent
An AI system capable of taking actions autonomously to achieve specific goals in an environment.
Algorithm
A set of rules or instructions designed to solve a specific problem or perform a task.
Artificial Intelligence (AI)
The simulation of human intelligence in machines programmed to think and learn like humans.
Inference
The process of a trained AI model generating a response or making a prediction based on new input.
Model
An AI model is a computer program built to work like a human brain, processing input to generate a response.