Inference
Using a trained AI model to generate outputs
What is Inference?
The process of running new inputs through a trained AI model to generate predictions, text, classifications, or other outputs. Training builds the model; inference uses it. Every time you send a prompt to ChatGPT, that's inference. Understanding inference matters because it's where costs accumulate, optimizing inference speed and efficiency is critical for production AI applications, especially at scale.
How it works
Key takeaways
-
Running the model on new inputs = inference
-
This is where your per-query cost lives
-
Latency and throughput are key metrics
Real-world example
Every time ChatGPT generates a response, that's inference. The model's weights are frozen, it's running a forward pass on your input. Inference is what costs money per query, and optimizing it is a whole discipline.
