Why is measuring codebase token counts important before using AI refactoring?

LLMs charge per million tokens and have finite context windows. Feeding uncompressed repositories into Claude 3.5 Sonnet or GPT-4o inflates API costs, degrades reasoning performance ('needle in a haystack' degradation), and can trigger silent truncation of critical source files.

How does Creview calculate code tokens?

Creview combines Tree-sitter AST syntax tree leaf parsing with BPE token estimation algorithms, giving you both the structural parse token count and the estimated LLM input token count across Python, JavaScript, TypeScript, HTML, CSS, and JSON.

How can I reduce codebase tokens before sending code to an LLM?

Effective techniques include extracting AST interfaces/stubs rather than full function bodies, stripping docstrings and inline comments, removing generated assets or vendor lockfiles, and using Creview to isolate the specific modules with high cyclomatic complexity.

LLM Context Budget • Developer Tool

Codebase Token Counter & Context Optimizer

Calculate exact token footprints, measure context window consumption across Claude 3.5 Sonnet, GPT-4o, and Gemini, and optimize your repository prompt budgets before AI-driven refactoring.

Source Code or Snippet

import asyncio
from typing import List, Dict, Any

class RepositoryContextManager:
    """Manages multi-file repository tokens before sending prompts to LLMs."""
    
    def __init__(self, model_name: str = "claude-3-5-sonnet-20241022", max_budget: int = 120000):
        self.model_name = model_name
        self.max_budget = max_budget
        self.cache: Dict[str, int] = {}
        
    async def filter_ast_nodes(self, file_content: str) -> str:
        # Strip comments, docstrings, and non-essential implementations
        lines = [line for line in file_content.splitlines() if not line.strip().startswith("#")]
        return "\n".join(lines)
        
    def estimate_cost(self, tokens: int, rate_per_million: float = 3.00) -> float:
        return (tokens / 1_000_000) * rate_per_million

Paste any file or module to update calculations live 0 characters

Context Budget Analysis Real-time

Estimated Input Tokens

Claude 3.5 Sonnet (200k limit) 0%

GPT-4o / GPT-4 Turbo (128k limit) 0%

Gemini 1.5 Pro (2M limit) 0%

Single-Call API Cost (Input)

Claude 3.5 Sonnet ($3.00/1M): $0.0000

GPT-4o ($2.50/1M): $0.0000

DeepSeek-V3 ($0.14/1M): $0.0000

How to Optimize Repository Tokens for AI Coding Agents

AI coding tools like Cursor, Claude Dev/Cline, and Windsurf rely on prompt context to generate high-quality code. Bloated context leads to degraded reasoning ("lost in the middle") and unnecessary cloud bills.

1. AST Interface Pruning

Instead of passing full function bodies across 50 files, extract only class signatures, type annotations, and docstrings. This preserves architectural knowledge while cutting 70-80% of tokens.

2. Eliminate Vendor & Build Artifacts

Lockfiles (`package-lock.json`, `poetry.lock`), build folders (`dist/`, `.next/`), and generated test snapshots can secretly blow through 100k+ tokens in a single agent request.

3. Target High Complexity Nodes First

Use Creview's cyclomatic complexity metrics to identify the specific 5% of functions causing 80% of bugs, feeding only those high-risk nodes to LLMs for focused refactoring.

Frequently Asked Questions

Why do high token counts harm LLM refactoring quality?

Research on long-context models shows that attention mechanisms degrade when searching for critical logic in massive prompts (the "needle-in-a-haystack" degradation). Smaller, structurally curated prompts consistently outperform raw whole-codebase dumps.

Can Creview measure tokens across an entire GitHub repository?

Yes. The full Creview application parses whole projects, calculating total token footprints per file and folder so you can plan agent budgets before initiating large refactoring sweeps.

Is code uploaded to third parties during token counting?

No. Calculations on this page happen directly inside your browser. No code is transmitted to external servers or AI providers.