Documentation Hub

ContextMiner API

Think of ContextMiner as an intelligent buffer between your code editor, agents, and upstream LLM providers via the universal Custom OpenAI API connector (OpenAI, Anthropic, Gemini, DeepSeek, Groq, Mistral, OpenRouter, and self-hosted models). It runs deterministic Structural Logic Analysis and local compliance scrubbing in under 5 milliseconds on the CPU—cutting token spend by 15% to 25% on the Free tier and up to 80%+ on Pro and Enterprise plans (60–80%+ in General Mode, 40–55%+ in IDE/Dev Mode), with zero prompt storage.

API Proxy Architecture

ContextMiner is a drop-in reverse proxy. By simply changing your provider base URL from api.openai.com/v1 to https://context-miner.com/v1, your prompt passes through a local pipeline that scrubs sensitive PII, prunes structural syntax deadweight via our Safe Syntax Parser, and forwards the clean payload directly to your AI model.

Step 1
Auth & Mode Routing
Validates cm_key, verifies quotas, and checks connection Mode (General vs IDE/Dev Mode).
Step 2
Compliance Shield
In General Mode, runs the local Compliance Shield to mask PII into Reversible Vault Tokens. In Dev Mode, bypasses masking for test data.
Step 3
Optimization Engine
Applies Structural Logic Analysis, Deterministic Payload Minification, and dynamic Semantic Relevance Filtering with calibrated integrity floors: 60%–80%+ in General Mode (20% floor) and 40%–55%+ in IDE/Dev Mode (45% floor protecting code and tool schemas).
Step 4
BYOK Provider Dispatch
Decrypts target API keys in-memory and routes payload to upstream model API.

Quick Start

Begin optimization by redirecting your requests. Select your programming environment below to view dependency setup and proxy code.

cURL
curl -X POST https://context-miner.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer cm_key_YOUR_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "Long HTML page or document here..."}
    ],
    "stream": false
  }'
15-80%+
Token reduction
<5ms
Latency overhead
BYOK
Secure Keys

API Reference

Interact with the proxy using standard parameters.

POST/v1/chat/completions

Proxy completions endpoint that intercepts payloads, strips data, and forwards optimized strings downstream.

• Base URL vs Endpoint URL

If you are building custom API scripts (Python, cURL, Node), use the full endpoint URL: https://context-miner.com/v1/chat/completions. However, if you are configuring an IDE like Cursor, Cline, or Roo (or libraries like LangChain), they automatically append the chat path. For these apps, set your Base URL exactly to: https://context-miner.com/v1 (without the trailing slash).

API Headers

Header KeyDefault ValuePurpose
Content-Typeapplication/jsonSpecifies input payload structure.
AuthorizationBearer cm_key_YOUR_API_KEYYour ContextMiner proxy API key.

Request Body Parameters

FieldTypeRequirementDescription
modelstringRequiredUpstream model identifier (e.g. gpt-4o, deepseek-chat).
messagesarrayRequiredList of chat message objects containing role and content.
streambooleanOptionalFully supported. Set to true to receive Server-Sent Events (SSE). The proxy translates Anthropic and Gemini stream formats to the OpenAI choices[].delta.content standard in real time.

• Media Block Policy

To ensure absolute data sovereignty, low latencies, and prevent unmaskable visual PII leakage, ContextMiner blocks incoming media formats including images, audio, and video arrays when running in General Mode. Sending messages containing image_url, audio, or video properties in this mode immediately returns an HTTP 415 Unsupported Media Type status code. In IDE/Dev Mode, media and multimodal files are fully allowed.

Response Schema

The proxy returns standard OpenAI structures. The `prompt_tokens` in usage reports the final optimized token counts, which yields actual billing reductions downstream.

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Optimized output response from your AI provider."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 1260,     // Post-compression prompt token count
    "completion_tokens": 100,  // Response token count
    "total_tokens": 1360
  }
}

HTTP Error Codes & Resolution

StatusNamePrimary CauseResolution
400Bad RequestMalformed JSON payload or incorrect fields.Check your parameters and ensure body is correct JSON.
401UnauthorizedMissing, empty, or invalid Bearer key.Validate your proxy API key (`cm_key_...`).
402Payment RequiredSubscription past due or unpaid invoice on Stripe.Update your payment method in the Billing tab to restore API access immediately.
409Key Conflict / Key Already in UseThe upstream AI provider key is already connected to another Free-tier account.Upgrade to Pro to share this key across multiple accounts, or connect a unique API key.
415Unsupported MediaPayload contains images, audio, or video while running in General Mode.Remove media blocks or switch your connection to IDE/Dev Mode in the dashboard.
429Quota Exceeded / Rate LimitedMonthly request limit reached (Free: 1,000, Pro: 100k, Enterprise: 500k).Upgrade plan, add on-demand request packs, or wait for monthly cycle renewal.
508Loop DetectedRecursive proxy call caught by the anti-loop header (x-contextminer-proxy).Ensure your upstream connection does not point back to ContextMiner recursively.
502 / 504Upstream Provider ErrorDownstream model API (OpenAI, Anthropic, Gemini) timed out or returned 5xx.Check your provider account status or retry with exponential backoff.

Content Types

ContextMiner identifies document types dynamically to strip formatting and markup without compromising structural integrity.

Formathtml

Prunes CSS styles, inline javascript scripts, structural header blocks, and footer wrappers. Keeps text structured via Semantic Skeleton Extraction.

Preserves: Form tags, links, data markers, table structures, and input shapes
Formatjson

Prunes null entries, duplicate maps, blank configuration entries, and trailing layout spacing.

Preserves: Root hierarchies, parameter settings, array keys, and config values
Formatcsv

Prunes consecutive empty lines, margins, duplicate headers, and guide rows.

Preserves: Column structures, floating point indices, and records timestamps
Formatlogs

Deduplicates verbose repeating stack loops and standardizes common log output shapes.

Preserves: Exception names, files lines, call trace ordering, and error logs
Formatcode (Structural Logic)

Processes programs through our Structural Logic Analyzer to discard comments, empty statements, and formatting.

Preserves: Flow logic, function arguments, import blocks, and scoping boundaries
Formatmarkdown

Normalizes header tags, collapses nested spacings, and strips decoration lines.

Preserves: Nested lists, table keys, links, quotes, and inline code segments

Structural Logic Analysis Comparison

Unlike raw text manipulation which can break syntax, the Structural Logic Analyzer understands code blocks. Comments, multi-line gaps, docstrings, and formatting margins are safely stripped. Logic remains fully compilable and semantically identical to the model.

Before Minification (100% readable, 23 lines)
/**
 * Custom hook to track intersection
 * Filters visible items on page
 */
export function useTracker(elementId) {
  const [visible, setVisible] = useState(false);
  const observerRef = useRef(null);

  useEffect(() => {
    // Check intersection support
    if (typeof window !== 'undefined') {
      observerRef.current = new IntersectionObserver(([e]) => {
        setVisible(e.isIntersecting);
        if (e.isIntersecting) {
          console.log("Element visible: " + elementId);
        }
      });
      const el = document.getElementById(elementId);
      if (el) observerRef.current.observe(el);
    }
    return () => observerRef.current?.disconnect();
  }, [elementId]);

  return visible;
}
After Structural Logic Analysis (Compressed, 1 line)
export function useTracker(elementId){const[visible,setVisible]=useState(false);const observerRef=useRef(null);useEffect(()=>{if(typeof window!=='undefined'){observerRef.current=new IntersectionObserver(([e])=>{setVisible(e.isIntersecting)});const el=document.getElementById(elementId);if(el)observerRef.current.observe(el)}return()=>observerRef.current?.disconnect()},[elementId]);return visible}

Engine Deep Dive

Behind the scenes, ContextMiner utilizes a dual-engine architecture to optimize prompts and balance data fidelity.

1

Structural Logic Analyzer & Deterministic Payload Minifier

Powered by the Dynamic Context Optimizer, the Structural Logic Analyzer loads programming syntax natively to strip comments, whitespaces, and unused libraries. Simultaneously, the Deterministic Payload Minifier specifically targets and collapses giant JSON payloads and structured trees. Upstream coding bots (e.g. Cursor) send highly condensed snippets without syntax breakage.

2

Reversible Context Compression (CCR) & Dynamic Context Optimizer

CCR identifies structural patterns and prompt prefix sequences in an in-memory cache. In IDE & Dev Mode (40–55%+ savings, 45% floor), it preserves code structure, function schemas, and local test endpoints. In General Mode (60–80%+ savings, 20% floor), dynamic Semantic Relevance Filtering and Web Noise Filtering compress large prompts without losing conversational coherence.

3

Selective Multimodal Filtering & Zero-Latency Gateway

In General Mode, non-text media files are filtered to maximize text token savings. In IDE & Dev Mode, all images (UI screenshots, mockups), PDFs, documents, and multimodal files accepted by upstream AIs are fully allowed and routed downstream.

4

Sub-millisecond Offline Tokenizer Heuristic

Evaluates input token volume locally in microseconds (<0.1ms) using model-calibrated heuristics, eliminating upstream count queries and network roundtrips.

Explore Offline Tokenizer & Cost Calibration Multipliers→

Estimated Savings Formulation

Savings are logged asynchronously. Input prompt tokens are priced based on the upstream cost definitions. The formulation is:

Savings (USD) = (Prompt Tokens Saved / 1,000,000) * Upstream Model Input Cost Per 1M Tokens

Offline Tokenizer & Cost Calibration

Sub-millisecond Lexical BPE Approximation

ContextMiner uses a lexical BPE approximation formula to estimate prompt token usage offline before sending it upstream. It matches official token counts with 98% accuracy by mapping operators, symbols, whitespaces, and long words.

Model-Specific Multipliers

Calibrated multipliers applied to normalize BPE estimates against proprietary upstream tokenizers.

98% Accuracy
Provider FamiliesToken MultiplierHeuristic Reason
OpenAI (GPT-4o, o1, o3)0.90xCalibrated for the efficient o200k_base vocabulary.
Anthropic (Claude 3.5 Sonnet)0.95xSlightly denser token mapping than generic cl100k.
DeepSeek (V3, R1)0.92xDeepSeek vocabulary optimizations.
Google Gemini (1.5 Pro/Flash)1.05xSentencePiece layout structures, slightly less dense on code blocks.

Real-time ROI & Savings Computation

Savings are computed instantly and recorded asynchronously without holding up streaming response chunks. Prompt tokens saved are converted directly into dollar amounts based on target provider model pricing:

Savings (USD) = (Prompt Tokens Saved / 1,000,000) * Upstream Model Input Cost Per 1M Tokens

Compliance Shield

Enterprise PII anonymization. Replaces sensitive customer data with standardized placeholders before it leaves your network.

Zero-Egress Security Model

All masking runs locally on CPU via fast NLP entity recognition and deterministic pattern detection. Thanks to our Token Vault technology, original data is fully reconstructed for your end-users.

Supported Global Frameworks

LGPD (Brazil)HIPAA (USA)CCPA (California)GDPR (Europe)UK DPA 2018

Compliance Framework Mapping

LGPD

LGPD (Brazil)

General Data Protection Law (LGPD)

Protects the personal information of Brazilian citizens. Masking automatically matches formatted and unformatted versions of CPF, CNPJ, RG, CNH, PIS, Voter Registration Card, credit cards, emails, and Brazilian phone numbers to align with data minimization principles.

HIPAA

HIPAA (USA)

Health Insurance Portability & Accountability Act

Protects health data (PHI) in the United States. Under Safe Harbor regulations, direct identifiers must be de-identified. ContextMiner masks Social Security Numbers (SSN), Medicare Beneficiary Identifiers (MBI), emails, and phone numbers.

CCPA

CCPA (California)

California Consumer Privacy Act

Grants privacy rights to California consumers. Automatically anonymizes network IDs like IP and MAC Addresses, personal identifiers such as SSNs, emails, and phone numbers to prevent downstream data mining.

GDPR

GDPR (Europe)

General Data Protection Regulation

Enforces data security for EU citizens. Automatically redacts bank coordinates like IBAN, credentials including Passport Numbers, emails, IP addresses, and MAC addresses.

UK DPA

UK DPA 2018

Data Protection Act 2018

UK implementation of GDPR principles. Identifies and redacts British identity document layouts including National Insurance Numbers (NIN), UK Driving Licenses, UK Passport formats, bank coordinates (IBAN), and email directories.

Masking Replacements Directory

Personal identifiers are replaced by Reversible Tokens (e.g. `{{TKN_1A2B}}`). Whether passed formatted (with dots, dashes, spaces, or slashes) or completely unformatted (e.g. raw digit strings), the shield identifies them and restores them upon response.

LGPD / Brazilian Identification Forms

CPF Formatted[CPF_MASKED]
Input: 123.456.789-00
Standard format (11 digits with dots & dash).
CPF Unformatted[CPF_MASKED]
Input: 12345678900
Matched via raw 11-digit parsing.
CNPJ Formatted[CNPJ_MASKED]
Input: 12.345.678/0001-00
Corporate tax ID with punctuation.
CNPJ Unformatted[CNPJ_MASKED]
Input: 12345678000100
Matched via raw 14-digit sequence.
RG Formatted[RG_MASKED]
Input: 12.345.678-9
Identity card layout with punctuation.
RG Unformatted[RG_MASKED]
Input: RG: 123456789
Identified via "RG" or "registro" prefix keys.
CNH ID Number[CNH_MASKED]
Input: CNH: 12345678901
Driver license (11 digits) matching contextual keys.
Voter ID Formatted[VOTER_MASKED]
Input: Título: 1234 567 890 12
12-digit voter card with spaces.
Voter ID Unformatted[VOTER_MASKED]
Input: Título: 123456789012
12 digits with "título" or "eleitor" prefix keys.
PIS Formatted[PIS_MASKED]
Input: 120.55555.21-9
Social identifier layout with formatting.
PIS Unformatted[PIS_MASKED]
Input: PIS: 12055555219
11 digits with "PIS" or "pasep" prefix keys.

HIPAA & CCPA / US Identification Forms

SSN Formatted[SSN_MASKED]
Input: 123-45-6789
Social Security Number with dashes.
SSN Unformatted[SSN_MASKED]
Input: 123456789
Raw 9 digits or identified via "SSN" context keys.
Medicare MBI[MEDICARE_MASKED]
Input: 1EG4-TE5-MK72
Medicare Beneficiary ID (11 chars, alphanumeric).
Medicare Unformatted[MEDICARE_MASKED]
Input: 1EG4TE5MK72
Raw MBI matched using alphanumeric heuristics.

GDPR & UK DPA / European Identifiers

IBAN Formatted[IBAN_MASKED]
Input: DE89 3704 0044 0532 0130 00
Bank account number with spacing.
IBAN Unformatted[IBAN_MASKED]
Input: DE89370400440532013000
Clean string matching standard country prefixes.
UK NIN[NIN_MASKED]
Input: QQ123456A
National Insurance format (2 letters, 6 digits, 1 letter).
UK Driving License[LICENSE_MASKED]
Input: JONES123456AB9CD
Licence codes containing name prefix + numbers.
UK/EU Passport[PASSPORT_MASKED]
Input: Passport: 123456789
9-digit sequence matching "passport" prefix keys.
EU Passport Alphanumeric[PASSPORT_MASKED]
Input: C12345678
1 letter followed by 8 digits (common EU layout).

General Contact & Network Coordinates

Email Address[EMAIL_MASKED]
Input: user@domain.com
Matches standard RFC 5322 structures.
Phone BR Formatted[PHONE_MASKED]
Input: +55 11 99999-9999
Includes country code, area code, and dash.
Phone BR Unformatted[PHONE_MASKED]
Input: 11999999999
11 digits starting with area code (11-99).
Phone US Formatted[PHONE_MASKED]
Input: +1 (555) 123-4567
Matches standard US country and area codes.
Phone US Unformatted[PHONE_MASKED]
Input: 5551234567
10 digits (matches standard US area code).
Credit Cards[CARD_MASKED]
Input: 4111 2222 3333 4444
Matches 13-16 digit cards (Visa, MasterCard, etc).
IP Address Dotted[IP_MASKED]
Input: 192.168.1.1
Standard IPv4 dotted decimals (loopback addresses 127.0.0.1 and 0.0.0.0 are preserved for development).
IP Address Unformattedip: [IP_MASKED]
Input: ip: 19216811
Integer arrays with "IP" or "endereco" prefix keys.
MAC Address[MAC_MASKED]
Input: 00:1A:2B:3C:4D:5E
12 hex digits with colons or dashes.

Credentials & Secrets (Critical Security)

JWT Token[JWT_MASKED]
Input: eyJhbGciOi...
Decoders matching header.payload.signature structures.
AWS Access Key[AWS_KEY_MASKED]
Input: AKIAIOSFODNN7...
16-char AWS access keys (excluding AWS test tokens).
RSA Private Key[PRIVATE_KEY_MASKED]
Input: -----BEGIN RSA PRIVATE KEY-----
PEM structures (RSA, DSA, EC, OPENSSH).
API Secrets / Keys[SECRET_MASKED]
Input: sk-proj-1234567...
Stripe, OpenAI, or Bearer keys with high entropy.

• Prompt Injection Guardrails & NLP

Beyond deterministic boundary safety, ContextMiner integrates a lightweight NLP NER engine to identify complex, unstructured entities (names, cities, organizations). It also implements Prompt Injection Guardrails to intercept Jailbreak attacks (e.g. "ignore all instructions") at the proxy level before they consume your API tokens.

Universal Custom OpenAI API Connector

Bring Your Own Key. ContextMiner standardizes on the Universal Custom OpenAI API Connector (OpenAI-Compatible protocol) to route downstream calls to any LLM provider or self-hosted engine with zero code changes.

OpenAI & OpenRouter

gpt-4o, gpt-4o-mini, o1, o3-mini, all OpenRouter models

DeepSeek & Groq

deepseek-chat, deepseek-reasoner, llama-3.3-70b-versatile

Gemini & Self-Hosted

gemini-1.5-pro (OpenAI endpoint), Ollama, vLLM, LM Studio

Universal OpenAI-Compatible Connection Specifications

Connection StandardTarget Provider Base URL ExampleAuthentication Header KeyAPI Schema Format
Custom OpenAI (OpenAI Official)https://api.openai.com/v1Authorization: Bearer [KEY]OpenAI Compatible
Custom OpenAI (OpenRouter)https://openrouter.ai/api/v1Authorization: Bearer [KEY]OpenAI Compatible
Custom OpenAI (DeepSeek)https://api.deepseek.comAuthorization: Bearer [KEY]OpenAI Compatible
Custom OpenAI (Groq)https://api.groq.com/openai/v1Authorization: Bearer [KEY]OpenAI Compatible
Custom OpenAI (Google Gemini)https://generativelanguage.googleapis.com/v1beta/openaiAuthorization: Bearer [KEY]OpenAI Compatible
Custom OpenAI (Ollama / vLLM)http://localhost:11434/v1 or http://localhost:8000/v1Authorization: Bearer [KEY]OpenAI Compatible

Agent & IDE Integrations

Forward coding assistant traffic through the local proxy.

1. Cursor IDE Setup

Cursor allows configuring custom OpenAI-compatible models. By setting the endpoint base to the ContextMiner proxy, it routes coding queries with ultra-low latency.

1. Open Cursor and navigate to Settings ➔ Models.
2. Under OpenAI API Key, input your proxy key: cm_key_YOUR_API_KEY.
3. Click on Override OpenAI Base URL and enter: https://context-miner.com/v1.
4. Enable the target model (e.g. gpt-4o) and test the chat.
• Pro Tip: Enable IDE & Dev Mode on your connection in API Keys & BYOK to protect code syntax, tool calls, and local test endpoints (127.0.0.1) with calibrated 40–55%+ savings.

2. Aider CLI Setup

Configure Aider to route prompt and codebase contexts through ContextMiner using environment variables.

Set the following variables in your terminal shell or place them in a .env file:
# Set model endpoint base export OPENAI_API_BASE="https://context-miner.com/v1" # Inject your proxy credential key export OPENAI_API_KEY="cm_key_YOUR_API_KEY" # Execute aider specifying target model aider --model gpt-4o
• Alternatively, save these in your global configuration file: ~/.aider.conf.yml.

3. Cline / Roo Code Assistants

VS Code extensions (Cline, Roo Code) supporting Custom OpenAI endpoints can be configured similarly.

1. Open the assistant configuration panel in VS Code.
2. Set Provider to OpenAI Compatible.
3. Set Base URL to: https://context-miner.com/v1.
4. Set API Key to your ContextMiner key: cm_key_YOUR_API_KEY.
• Note for Autonomous Agents: Turn ON IDE & Dev Mode on your connection to ensure function calling tools, JSON arguments, and multi-file diffs run with 100% integrity.

Fair Use & Anti-Abuse Policy

View Pricing & Plans →

To protect platform uptime and ensure high-speed processing for all developers and production systems, ContextMiner enforces transparent fair use guardrails and anti-abuse policies.

Fair Use & Anti-Abuse Guardrails

To protect our servers from scripted loops, preserve low latency, and keep the service reliable, we enforce four core guardrails:

1One Active Account per IP Address

We allow 1 active free account per IP network. If you need multiple developers or agents on the same company network, upgrade to Pro or reach out to our team for custom provisioning.

2Unique Upstream Key (BYOK Fingerprinting)

Each upstream AI provider key (OpenAI, Anthropic, Gemini) can be connected to one Free account at a time. We calculate an irreversible SHA-256 hash of your key upon saving—your raw secrets are never logged or exposed in plain text. Pro and Enterprise accounts have no key reuse limits.

3Verified Personal & Work Emails

Temporary or disposable email domains (such as 10minutemail or tempmail) are blocked during registration. We normalize email aliases so accounts stay clean and genuine.

4Zero-Egress Security Standard

We record strictly numerical token counts and byte metrics in our database. Your code snippets, system prompts, chat completions, and API keys are never written to disk or used for model training.