How to Measure Token Usage and Cost per Request

Introduction to Token Usage and Cost Measurement

When building applications powered by Large Language Models (LLMs) like OpenAI's GPT series, understanding how you are billed is crucial. LLMs do not charge by the request, the hour, or the word. Instead, they charge by the token. A token is a chunk of text—often a word or a part of a word—that the model reads (input/prompt tokens) and generates (output/completion tokens).

Measuring token usage and calculating the cost per request matters for several reasons. First, it allows you to forecast and control your operational expenses, preventing unexpected billing surprises. Second, it helps you identify inefficiencies in your prompts. If a simple query is consuming thousands of tokens, you can optimize your system prompt to be more concise. Finally, tracking costs per user or per feature enables you to implement usage limits and build sustainable, profitable AI-driven products.

Understanding Token Usage in API Responses

Most modern LLM APIs return metadata about token consumption alongside the generated text. Typically, this metadata is found in a usage object within the API response. This object breaks down the consumption into three main categories:

Prompt Tokens: The number of tokens consumed by your input, including the system prompt, user message, and any conversation history.
Completion Tokens: The number of tokens generated by the model in its response.
Total Tokens: The sum of prompt and completion tokens.

Because providers usually price input and output tokens differently (with output tokens generally being more expensive), you must track them separately to calculate costs accurately.

How to Measure Token Usage per Request

To measure token usage, you need to extract the usage data directly from the API response object. Below is a practical example using Python and the OpenAI API. This example sends a simple chat completion request and prints out the token metrics returned by the server.

Example: Measuring Tokens with OpenAI API (Python)

import openai

# Ensure your API key is set in your environment variables
# openai.api_key = "your-api-key"

def get_chat_completion_and_usage(user_message):
    try:
        response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": user_message}
            ]
        )
        
        # Extract the usage object from the response
        usage = response['usage']
        
        # Extract the generated text
        answer = response['choices'][0]['message']['content']
        
        return answer, usage
        
    except Exception as e:
        print(f"An error occurred: {e}")
        return None, None

# Execute the function
user_query = "Explain quantum computing in one sentence."
response_text, token_usage = get_chat_completion_and_usage(user_query)

if token_usage:
    print(f"Response: {response_text}\n")
    print("--- Token Usage ---")
    print(f"Prompt Tokens:     {token_usage['prompt_tokens']}")
    print(f"Completion Tokens: {token_usage['completion_tokens']}")
    print(f"Total Tokens:      {token_usage['total_tokens']}")

Calculating Cost per Request

Once you have the token usage, calculating the cost is a simple math operation. You need to maintain a pricing dictionary that reflects the current rates charged by your LLM provider. You then multiply the prompt tokens by the input rate and the completion tokens by the output rate.

Example: Calculating Cost in Python

# Pricing per 1,000 tokens (hypothetical rates for demonstration)
PRICING_TABLE = {
    "gpt-3.5-turbo": {
        "prompt": 0.0015,
        "completion": 0.002
    },
    "gpt-4": {
        "prompt": 0.03,
        "completion": 0.06
    }
}

def calculate_request_cost(model, prompt_tokens, completion_tokens):
    # Retrieve the pricing for the specified model
    model_pricing = PRICING_TABLE.get(model)
    
    if not model_pricing:
        raise ValueError(f"Pricing for model '{model}' not found.")
    
    # Calculate individual costs
    prompt_cost = (prompt_tokens / 1000) * model_pricing["prompt"]
    completion_cost = (completion_tokens / 1000) * model_pricing["completion"]
    
    # Return the total cost
    return prompt_cost + completion_cost

# Assuming token_usage is the dictionary retrieved from the previous example
if token_usage:
    model_used = "gpt-3.5-turbo"
    cost = calculate_request_cost(
        model=model_used,
        prompt_tokens=token_usage['prompt_tokens'],
        completion_tokens=token_usage['completion_tokens']
    )
    
    print(f"\n--- Cost Calculation ---")
    print(f"Model used: {model_used}")
    print(f"Total Cost: ${cost:.6f}")

Best Practices for Token and Cost Management

Simply measuring costs is only half the battle. To build efficient applications, you should actively work to reduce unnecessary token consumption. Here are several best practices to follow:

Optimize System Prompts: Keep your system instructions concise. Remove redundant explanations or overly verbose formatting instructions. Every token in a system prompt is billed on every single request.
Manage Conversation History: In chat applications, sending the entire conversation history with every new message will cause token usage to scale quadratically. Implement sliding windows or summarization techniques to cap the history length.
Set Max Tokens: Always use the max_tokens parameter (or equivalent) to limit the length of the model's response. This prevents a runaway model from generating thousands of unnecessary tokens if it misunderstands the prompt.
Implement Caching: If users frequently ask identical questions, cache the responses. Serving from a cache costs zero tokens and reduces latency.
Choose the Right Model: Do not use a heavy, expensive model like GPT-4 for simple tasks like text classification or formatting. Route simpler tasks to cheaper, faster models like GPT-3.5-turbo.
Log and Monitor: Store the token usage and calculated cost in your database alongside the user ID and request timestamp. This allows you to build dashboards to track cost trends over time and identify power users who might need rate limits.

Conclusion

Measuring token usage and calculating the cost per request is a fundamental skill for any developer working with Large Language Models. By extracting the usage data from API responses and applying a simple pricing formula, you gain complete visibility into your application's operational expenses. Combined with proactive optimization strategies like prompt refinement, history management, and intelligent model routing, this practice ensures that your AI features remain both highly performant and financially sustainable as your user base grows.