Home/Blog/Architecture Deep-Dive
Architecture Deep-Dive6 min readNovember 10, 2026

Semantic Skeleton Extractor: How We Strip 76% HTML Noise in 1.8ms

Deterministic Web Noise Filtering and Semantic Skeleton Extraction for Agentic Web Crawling

CM
ContextMiner Engineering
Core Gateway Systems
Executive Summary & Key Takeaways
  • Over 75% of raw HTML tokens swallowed by AI crawlers are visual scaffolding: tracking pixels, Tailwind class strings, and cookie banner wrappers.
  • Headless browser renderers add 400ms+ of latency, while naive regex parsers regularly destroy tables and code snippets.
  • ContextMiner runs a single-pass linear extractor in 1.8ms directly at the gateway, producing clean semantic markdown ready for LLM attention heads.
  • Across 1,000 real-world documentation and commerce sites, prompt token consumption dropped 76.4% with 99.8% entity recall.

Web Payload Compression Benchmark (Top 100 Enterprise Docs & News Sites)

MetricRaw Scraped HTMLGeneric Markdown ConverterContextMiner Semantic Skeleton
Average Payload Size184.2 KB62.4 KB14.8 KB
Average Token Count46,050 tokens15,600 tokens3,700 tokens
Processing Latency0 ms (raw)48.2 ms1.8 ms
Cost per 1,000 Ingests (GPT-4o)$115.12$39.00$9.25
Entity Extraction Accuracy94.2%96.1%99.8%

1. The Real Cost of Scraped HTML in Agent Prompts

If you have ever inspected the raw HTTP payload coming out of an agentic web scraper, the numbers are sobering. A standard 1,800-word documentation page or technical blog post regularly weighs over 250 kilobytes. Almost none of that weight is the actual text your agent needs. It is markup built for web browsers: - Endless Tailwind utility class strings repeated across thousands of tags. - Embedded SVG icons that consume 150 to 400 tokens each just to render a tiny chevron. - Hydration state JSON blobs, cookie consent banners, and tracking scripts. - Container `<div>` wrappers nested ten or twelve levels deep with zero semantic signal. When an autonomous agent feeds that raw HTML straight to GPT-4o or Claude 3.5 Sonnet, you are paying frontier model rates for visual noise. At $2.50 per million input tokens, a 45,000-token page costs roughly $0.11 per fetch. Run that through a continuous crawler scraping 2,000 URLs a day, and you are burning hundreds of dollars each week on HTML boilerplate. Even worse: all that noise dilutes the model's self-attention. The model has to sift through thousands of CSS classes just to locate the two numbers you actually asked it to extract.

Attention Dilution in Long Prompts

Longer prompts do not just cost more money; they increase retrieval errors. When 80% of a prompt consists of repetitive DOM scaffolding, models suffer measurable drops in needle-in-a-haystack accuracy.

2. Under the Hood: The 1.8ms Extraction Pipeline

Most engineering teams tackle this problem in one of two ways, both flawed. Option one is spinning up Puppeteer or Playwright to run reader-mode extraction. It produces clean text, but the penalty is brutal: 400ms to 900ms of latency per page and heavy Chromium container overhead. Option two is writing quick regex rules. That is fast, but it is fragile. Naive regex cleaners routinely butcher markdown code blocks, drop table headers, and mangle nested bullet points. The Semantic Skeleton Extractor takes a different route. Built directly into ContextMiner's gateway runtime, it processes the raw HTML stream in a single linear pass averaging 1.8 milliseconds. Here is what happens during that pass: - **Structural Tree Pruning:** Elements that cannot contain relevant text—like `<script>`, `<style>`, `<svg>`, `<noscript>`, and tracking wrappers—are dropped upfront before full parsing. - **Attribute Sanitization:** Attributes like `class`, `style`, and `data-*` are stripped completely. The engine keeps only structural anchors: `href`, `alt`, and table-alignment attributes. - **Container Collapse:** Deeply nested layout wrappers are flattened into clean hierarchical markdown structures. Headers, lists, code fences, and data tables stay intact. The result is a clean semantic outline. The model gets exactly what it needs to understand the content, and nothing else.
crawler-agent.ts
import OpenAI from 'openai';

// Point your existing SDK to the ContextMiner Gateway
const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: 'https://proxy.context-miner.com/v1',
  defaultHeaders: {
    'x-contextminer-key': process.env.CONTEXTMINER_PROXY_KEY,
    // Turn on the Semantic Skeleton Extractor for scraped web content
    'x-cm-web-filter': 'aggressive',
  },
});

// A prompt with 220KB of messy scraped HTML gets compressed
// to ~16KB of clean markdown at the gateway before hitting OpenAI
const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: 'Extract key pricing tiers and feature limits from this page.' },
    { role: 'user', content: rawScrapedHtmlPayload },
  ],
});

3. Benchmarking 1,000 Real-World Pages

The critical question with any compression system is fidelity. Did the filter drop something important? To test this, we built a benchmark harness across 1,000 production pages from developer docs (AWS, Stripe, GitHub), financial filings (SEC EDGAR), and news sites. We then ran structured entity extraction tasks against ground truth. The metrics: - **Entity Recall:** 99.8%. Across product specs, financial figures, dates, and names, the extracted skeletons retained the exact data points present in the raw HTML. - **Latency:** An average of 1.8ms per page, compared to 48ms for standard Node-based markdown converters. - **Token Reduction:** A 76.4% reduction in prompt tokens across the dataset. Because the model receives focused, high-density text rather than miles of layout wrappers, downstream reasoning is sharper. You get faster responses, lower token bills, and fewer hallucinations.

4. Setup: Zero SDK Changes Required

You do not need to install local scraper libraries or rewrite your ingestion workers. ContextMiner functions as an OpenAI-compatible reverse proxy. You change your SDK's baseURL, supply your unified proxy key, and keep using your existing code. When your crawler sends a prompt containing scraped web markup, the gateway detects the HTML, runs the Semantic Skeleton Extractor in memory, and forwards the clean payload upstream. Nothing is logged to disk.
Try the Gateway Free

Optimize Your Prompts in 2 Minutes

ContextMiner is a drop-in OpenAI-compatible proxy. Connect your LLM API keys and instantly apply Semantic Skeleton Extraction, Algorithmic Relevance Filtering, and Zero-Egress PII Guardrails.