Turn any URL into structured intelligence.
Extract SEO, technologies, contacts, products, social profiles, security signals, and AI-ready content from any public website โ with one API call.
Free to try. No signup required for the demo.
๐ฎ Try the Live Demo โข ๐ Get Free API Key on RapidAPI โข ๐ API Documentation โข โก Code Examples
Paste a URL, get real results in seconds โ rapidapi-metadata-extractor.onrender.com runs the actual pipeline live, no API key needed. Try your own site, or one of these:
/demo/extractis the same no-key route the live demo page above uses (see "Honest Limitations" โ it's rate-limited per IP, not a documented/stable API endpoint). For integrating in code, use/api/v1/extractwith a RapidAPI key as shown below.
Or call it directly (swap in your key from RapidAPI for production use):
curl "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=github.com" \
-H "X-RapidAPI-Key: YOUR_KEY" -H "X-RapidAPI-Host: web-metadata-and-contact-extractor.p.rapidapi.com"
๐ฏ Use Cases
| Use case | What you get | |
|---|---|---|
| ๐งฒ | Lead generation | Find companies on a given tech stack (e.g. Shopify) with a weak SEO score, then prioritize outreach. |
| ๐ค | AI agents | Feed markdown_content and structured JSON straight into an agent or RAG pipeline as clean, LLM-ready context for any URL. |
| ๐ | SEO audits | Run the 14-point on-page score plus graded security headers across a URL list, on a schedule. |
| ๐ | Ecommerce | Pull product name/price/currency/availability/brand from Schema.org, OpenGraph, and Microdata โ cross-checked for conflicts. |
| ๐ต๏ธ | Competitive intelligence | Track a competitor's tech stack, metadata, and product data over time by re-checking the same URL list. |
Example: lead generation โ flag Shopify sites with a weak SEO score
import requests
RAPIDAPI_KEY = "YOUR_RAPIDAPI_KEY"
HEADERS = {
"X-RapidAPI-Key": RAPIDAPI_KEY,
"X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com",
}
BASE = "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract"
for url in ["shop-a.com", "shop-b.com", "shop-c.com"]:
data = requests.get(BASE, headers=HEADERS, params={"url": url}).json()
if "Shopify" in data["detected_technologies"] and data["seo_score_percentage"] < 60:
print(f"{url}: Shopify + {data['seo_score_percentage']}% SEO -> outreach candidate")
Example: AI agent tool call โ clean Markdown for any URL
import requests
def read_url_as_markdown(url: str) -> str:
"""Tool function an LLM agent can call to read any public webpage."""
resp = requests.get(
"https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/markdown",
headers={"X-RapidAPI-Key": "YOUR_KEY", "X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com"},
params={"url": url},
).json()
return resp["markdown_content"]
๐ Key Features
| Feature | Detail |
|---|---|
| โก Ultra-Fast Performance | ~150โ300ms live fetch. Powered by selectolax (C-Lexbor parser), Rust ORJSON, uvloop, HTTP/2 multiplexing, async DNS (non-blocking), 5-min DNS TTL cache. |
| ๐ง Adaptive SPA Byte Limit | Auto-detects React, Next.js, Vue, Angular, Nuxt, Svelte, Gatsby, Remix, Astro and expands download to 256 KB for richer data extraction. Static sites stay at 64 KB. |
| ๐ก๏ธ IP-Pinned Anti-SSRF Shield | DNS resolved once, IP validated against private/loopback/cloud-metadata ranges, connection pinned to IP with TLS SNI. Eliminates DNS Rebinding & Redirect SSRF. |
| ๐ฏ Rich SEO & OpenGraph Metadata | Title, Description, OG Image, OG Type, OG URL, OG Video, Favicon, Canonical URL, Language, Author, Theme Color, Robots directive, hreflang tags, H1 headings, image count. |
| ๐ง Contact Extractor | Public emails and phone numbers with smart DOM cleaning to eliminate false positives. Phones are additionally normalized (phone_details) via libphonenumber โ E.164 format, ISO country code, and possible/valid flags โ for numbers that include a country code; numbers without one aren't guessed. |
| ๐ฒ Social Profile Finder | Auto-detects 20 platforms: Twitter/X, LinkedIn, Facebook, Instagram, GitHub, YouTube, Telegram, TikTok, Threads, Bluesky, Mastodon, Discord, Reddit, Pinterest, Medium, GitLab, Vimeo, Dribbble, Behance, Snapchat. |
| ๐ ๏ธ 40+ Tech Stack Detector | WordPress, Shopify, WooCommerce, Webflow, React, Next.js, Vue, Angular, Svelte, TailwindCSS, Stripe, GA4, and more across CMS/ecommerce, frameworks, and analytics/payment/hosting categories. |
| ๐ฆ Schema.org JSON-LD + Product Parser | Parses all structured data schemas AND auto-extracts Product price, currency, availability, brand, rating and review count. |
| ๐ฏ Multi-Source Data Quality & Conflict Detection | Product price/currency/availability/brand are cross-checked across three distinct structured encodings โ JSON-LD, OpenGraph's product extension, and schema.org Microdata. product_field_confidence reports a per-field confidence score (more of these encodings agreeing = higher confidence, same philosophy as tech-stack detection) and which source won. When they genuinely disagree, a SOURCE_CONFLICT warning names the field, every value found, and which one was chosen and why. A top-level quality object (score, rendered: false, sources_used, warnings) summarizes response trustworthiness for programmatic consumers. Caveat: these three encodings often come from the same underlying product record on a given site, so agreement means "internally consistent," not "independently verified" โ see Honest Limitations below. |
| ๐ On-Page SEO Health Audit Score | 14-point automated on-page technical SEO diagnostic (0โ100%) with actionable warnings list, plus a structured seo_checks breakdown (check id, passed, severity, evidence) for programmatic use. |
| ๐ Internal vs External Link Classifier | Categorizes up to 100 hyperlinks per page. |
| ๐ค AI & LLM Clean Markdown Reader | Converts article text to clean Markdown for ChatGPT, Claude, RAG, and AI agents. Includes word count and reading time. |
| ๐ก RSS / Atom Feed Discovery | Auto-discovers RSS and Atom feed URLs. |
| ๐ Graded Security Headers Audit | HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer Policy, Permissions Policy โ each graded missing/weak/reasonable/strong from its actual value (e.g. a CSP with unsafe-inline scores as weak, not as "present = full credit"), not just presence, rolled into the percentage score. /api/v1/security additionally returns security_header_grades with the per-header breakdown. |
| ๐ 15-Min In-Memory Cache | Cached responses served in < 0.01 ms server-side processing time. |
| โก Multi-Worker + Auto-Reconnect Redis | uvicorn --workers N multi-process cluster (set via the WORKERS env var; gunicorn is a listed dependency for anyone who prefers it as the process manager, but the shipped Dockerfile launches uvicorn directly). Distributed rate limiting via Redis with startup ping validation, automatic reconnect every 30s, and immediate degraded_fallback status propagation to /health/details. |
๐ Split /health + /health/details | Minimal public liveness probe (/health). Full operational details (Redis mode, status, engine) secured via HEALTH_DETAILS_SECRET header on /health/details. |
| ๐งฏ Per-Host Circuit Breaker | Trips after repeated timeouts/connection failures/5xx to one host, fails fast during the cooldown window instead of burning the full connect+read timeout budget on every request, and probes recovery automatically (never trips on ordinary 4xx). |
| ๐งฉ Request Single-Flight + DNS Coalescing | Concurrent requests for the same URL (or the same hostname's DNS lookup) share one in-flight fetch instead of hammering the origin N times. |
| ๐ฉน Negative-Result Cache | Short-TTL caching of recent upstream failures (DNS/timeout/5xx) so a broken target fails fast instead of repeating the same slow failure for every request during an outage. |
| ๐งฑ Adaptive Byte-Limit Hardening | Streaming byte cap is enforced on the decoded chunk, closing a decompression-bomb gap where a small gzip/br payload could otherwise expand to megabytes in a single read. Response headers are also count- and size-bounded. |
| ๐ท๏ธ Structured Error Codes | Every error response carries a machine-readable error.code / error.retryable object (e.g. SSRF_BLOCKED, UPSTREAM_TIMEOUT, CIRCUIT_OPEN) alongside the existing detail string โ fully additive, v1 contract unchanged. |
| ๐ Prometheus Metrics + Structured JSON Logs | /metrics exposes request/cache/SSRF/circuit-breaker/rate-limit counters and latency histograms. Every log line is a JSON object with request_id for end-to-end tracing; known-sensitive fields are redacted automatically. |
| ๐ง Confidence-Scored Tech Detection | /api/v1/tech-stack now also returns technology_details: per-technology confidence score, matched evidence, and category (cms/ecommerce/framework/analytics/payment/hosting/โฆ), alongside the original flat list. |
| ๐ฆ Deeper Product & JSON-LD Parsing | Traverses @graph and top-level JSON-LD arrays (not just top-level objects), extracting SKU, MPN, GTIN/ISBN, seller, condition, price range, and images. |
| ๐ Unicode-Aware SEO & Keywords | Keyword extraction now matches non-ASCII scripts correctly (accented/Cyrillic/etc. content), and the SEO audit adds lang attribute, viewport, noindex, multi-H1, Twitter Card, and structured-data checks. |
| โก Lazy Extraction Per Endpoint | Specialized endpoints only run the extractors they actually need instead of the full pipeline โ /security never even parses the HTML tree, /tech-stack skips links/metadata/markdown/SEO entirely. /api/v1/extract runs the complete pipeline by default, but ?fields= now maps to the same narrow extraction as the matching specialized endpoint (fields=detected_technologies costs exactly what /tech-stack costs) instead of computing everything and filtering the response afterward. Results for different endpoints (or fields= combinations) hitting the same URL share one upstream fetch and accumulate into the same cache entry. |
| ๐ Mastodon Social Detection | Best-effort detection of the largest public Mastodon instances (mastodon.social, fosstodon.org, hachyderm.io, ...) alongside the existing 20 platforms โ decentralization means a hostname map can't cover every self-hosted instance, so this is intentionally partial rather than a false-positive risk. |
| ๐ฆ Rate-Limit Response Headers | Every response (success or 429) now carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, so clients can back off proactively instead of learning the limit by hitting 429. |
| ๐ก๏ธ Bot-Protection Detection | Cheap heuristic signature check (no extra requests, no JS execution) flags when the fetched page is a Cloudflare/Akamai/PerimeterX/CAPTCHA challenge page rather than real content โ either as bot_protection_detected: true on a normal response, or as a specific BOT_PROTECTION_DETECTED error code (instead of a generic upstream error) when the challenge is served with a 403/429/503 status. Distinguishes "the target blocked this request" from "the target genuinely has no title/description" or "a real outage". |
๐ฏ Performance SLA & Technical Architecture Notes
[!NOTE]
- Latency & Performance SLA: Server-side processing overhead (DOM cleaning, C-Lexbor parsing, Rust serialization) averages < 5ms. Live execution times depend on the target website's network latency and origin server response time. Repeating requests for the same URL hit the in-memory cache and return in < 0.01ms.
- IPv6 & IPv4-Mapped SSRF Shield: The Anti-SSRF validation engine enforces strict resolution checks across both IPv4 and IPv6, blocking loopback (
127.0.0.1,::1), link-local (169.254.169.254,fe80::/10), and IPv4-mapped IPv6 (::ffff:127.0.0.1) addresses.- Redis Rate Limiter (Fixed-Window, Auto-Reconnect): When
REDIS_URLis configured, rate limiting is distributed across all workers via a single atomic Redis Lua script (INCR+ first-hitEXPIREin one round-trip โ no window where a dropped connection could leave a counter with no TTL). Redis is pinged at startup and re-validated every 30 seconds. On failure, the service falls back to per-process TTLCache immediately and marksredis_status: degraded_fallbackin/health/details.- Split Health Endpoints:
/healthreturns a minimal public liveness payload./health/detailsreturns full operational status (Redis mode, trust_proxy, engine) and requires theX-Health-Secretheader whenHEALTH_DETAILS_SECRETenv var is set.- Horizontal Scaling: Single-instance โ in-memory TTLCache (60 req/min/IP). Multi-worker โ distributed Redis. Enterprise scale โ RapidAPI Gateway or Nginx.
- Zero-Trust Self-Hosting: Full Dockerfile (uvicorn multi-worker) and test suite included for self-hosted production deployments.
- Modular Codebase: The service is organized as an
app/package (security/,fetcher/,cache/,ratelimit/,extraction/,observability/,api/) rather than a single file โmain.pyis a thin backward-compatibility shim souvicorn main:app/gunicorn main:appkeep working unchanged.- Validated Configuration:
app/config.pyis backed bypydantic-settingsโ every tunable (rate limits, cache sizes/TTLs, byte-fetch limits, header limits, ...) is validated at startup with the exact same defaults as before, and now opportunistically overridable via env vars. An invalid value (negative sizes,STREAM_SOFT_LIMITaboveSTREAM_HARD_LIMIT, ...) fails immediately with a clear error instead of surfacing as a runtime bug later.- Lazy Extraction:
/api/v1/extractwith nofields=runs the full pipeline. Specialized endpoints โ and/api/v1/extract?fields=...โ instead run only their required extractor groups โ e.g./api/v1/securityskips HTML parsing entirely (headers-only),/api/v1/tech-stackskips metadata/links/markdown/SEO. Calling two different specialized endpoints (or field selections) for the same URL shares one upstream fetch and merges into one cache entry.- Tech Signature Prefiltering: Each of the 40+ technology signatures is gated by a cheap substring check against a once-lowercased copy of the page before its regex ever runs, instead of running every regex unconditionally โ same detection output, substantially less CPU on pages with many
<script>tags.- Observability:
GET /metricsexposes Prometheus counters and latency histograms (requests, cache hit/miss, SSRF blocks, circuit-breaker trips, rate limiting, bytes downloaded, etc.) โ open by default (standard Prometheus practice) but gate-able behindMETRICS_SECRET/X-Metrics-Secretif you're self-hosting with the port exposed to the internet and no reverse-proxy rule in front of it. All application logs are single-line JSON with arequest_idshared with theX-Request-IDresponse header, for correlating a request across logs and metrics.- Property-Based Fuzz Testing: Hypothesis-driven tests generate hundreds of adversarial inputs per run against URL normalization, the SSRF IP-safety classification (every IPv4/IPv6 address, not just a fixed list), and every extractor that runs on attacker-controlled page content โ asserting they never crash and the SSRF logic never lets a dangerous address through. This found and fixed two real bugs: a crafted
url=[query string could trigger an unhandled 500, and the embedded-credentials check (user:pass@host) could be bypassed by omitting the URL scheme.- Real-World Accuracy Benchmark:
benchmarks/bench_accuracy.pyruns the real pipeline against a hand-curated, independently-verified sample of 29 live URLs across ecommerce/news/blogs/SaaS/SPA/WordPress/multilingual/small/bot-protected sites โ seebenchmarks/README_ACCURACY.mdfor methodology and honest limitations. The first real run scored 61%: Brotli-compressed responses (common on modern CDN-fronted sites) were silently passed through undecoded and force-decoded as UTF-8, with no error raised anywhere โapp/fetcher/client.pyadvertisedAccept-Encoding: brbut the decoder package wasn't installed. Addingbrotlitorequirements.txtfixed it with no other code change; the same 29-URL run now scores 100%, 0 unexpected errors.
๐ฏ What This API Is โ And Isn't โ For
This API turns a URL into structured knowledge about that page: SEO/OpenGraph metadata, public contact signals, social profiles, tech stack, Schema.org/product data, graded security headers, link classification, and clean Markdown โ consolidated into one fast HTTP request. It's a metadata/intelligence layer over ordinary server-rendered HTML, not a general-purpose scraping tool or a browser-automation replacement. Keep that framing in mind when deciding whether it fits your use case.
โ Good fit:
- Company/website analysis โ pull tech stack, CMS, socials, contact signals, and SEO health for a given site in one call.
- Input for a sales/outreach workflow โ
/api/v1/contactsfinds public emails/phones/social links present on a page; it's a raw signal you feed into a lead-gen process you build (company identification, role verification, CRM enrichment), not a lead-enrichment product by itself โ see "Honest Limitations" below. - Automated SEO audits โ the 14-point on-page score, structured
seo_checksbreakdown, and graded security-header audit are built for exactly this. - Competitive/tech-stack monitoring โ compare tech stack, metadata, or product data across a known set of URLs over time. You supply and re-check the URL list; there's no built-in crawler (see below).
- AI agents / RAG pipelines โ
markdown_contentand the structured JSON fields are designed as clean LLM input, not raw HTML soup. - E-commerce data extraction โ product name/price/currency/availability/brand/SKU/reviews where a site exposes Schema.org JSON-LD, OpenGraph product tags, or Microdata, cross-checked with a per-field confidence score. Product responses use a short 3-minute cache specifically because price data goes stale faster than everything else.
- Link previews โ title, description, image, favicon for chat apps, social cards, and bookmarking tools.
โ Not a good fit:
- Complex/interactive scraping โ logins, multi-step forms, button clicks, infinite scroll, CAPTCHA-gated content. This API issues one HTTP GET and parses the HTML it gets back; it does not drive a browser or simulate user interaction.
- Heavily JS-rendered sites โ if the meaningful content only exists after client-side hydration, this API sees close to the same empty shell a plain HTTP client would. The adaptive byte limit improves coverage of what the server already sent; it doesn't execute JavaScript.
- Mass or distributed crawling โ there's no queue, frontier, or "give me a domain and I'll find its pages" mode. It's one URL in, one page's data out, by design โ kept stateless and fast rather than growing into a crawler.
- Penetration testing or vulnerability scanning โ the security-header audit grades the presence and quality of headers like HSTS, CSP, and X-Frame-Options. It does not probe for vulnerabilities, scan for misconfigurations beyond those headers, or constitute a security assessment. A high
security_scoremeans "this page sends good security headers," not "this site is secure." - "Extract anything from any site" scraping โ this is a fixed set of well-defined extractors (SEO, contacts, socials, tech, product, links, Markdown), not a programmable scraper for arbitrary custom fields or page structures.
For the technical reasoning behind these boundaries โ why JS execution, crawling, and bot-protection bypass are out of scope, and what the confidence/quality scores do and don't mean โ see "Honest Limitations" below.
โ ๏ธ Honest Limitations
This API fetches raw HTTP responses and parses HTML โ it is not a browser. That's a deliberate trade-off for speed (no browser startup, no JS execution wait, minimal memory per request), and it comes with real ceilings that no amount of additional regex or extractors can fully remove:
- No JavaScript execution. Content that only exists after client-side rendering (many SPAs, some login-gated pages, content behind "load more" interactions) will be missing or incomplete. A static/SSR page (most blogs, e-commerce product pages, marketing sites, GitHub, Wikipedia) works great; a client-rendered single-page app whose initial HTML is just
<div id="root"></div>will return mostly empty results โ the API cannot execute the JS that would fill it in. - Detecting a JS framework doesn't mean the JS problem is solved. The adaptive byte limit (see below) expanding to 256 KB when React/Next.js/Vue/etc. signatures are spotted improves coverage of the initial HTML โ it does not run a browser, execute hydration, or wait for client-side API calls.
Next.js -> more bytes -> a bigger parseis notNext.js -> Chromium -> JS execution -> hydration -> final DOM. Treat it as "we tried harder to read what the server already sent," not "we solved SPAs." - Bot-protection / CAPTCHAs are detected, not bypassed.
bot_protection_detected(see above) tells you that Cloudflare/Akamai/PerimeterX/a CAPTCHA blocked the request โ it does not solve the challenge. There is no workaround for this short of running a real browser. - Extraction is heuristic evidence, not ground truth โ including when it looks confident. Technology detection, contact info, social links, and product data are pattern/signature-based. A tech signature match (e.g. one CDN hostname in an image URL) proves the string was present somewhere the fetcher looked, not that the technology is meaningfully in use โ it could be dead code, a shared/third-party asset, a transitive dependency, or an accidental substring.
technology_details.confidencereflects how many independent signals agreed, not a probability that the detection is correct. See the live GitHub example above, where"Contentful"is detected at0.75confidence from a single CDN hostname in anog:imageURL โ plausible evidence, not proof GitHub runs on Contentful. - A
quality.scorenear 1.0 means the evidence available to this API was internally consistent โ not that the data is verified accurate. When JSON-LD, OpenGraph, and Microdata all report the same price, that's three encodings, not necessarily three independent sources โ a site frequently generates all three from the same underlying product record, so agreement can mean "one data source, expressed three ways" rather than three independently-verified facts. Readconfidence: 0.98as "nothing on the page contradicts this," not "there's a 98% chance this is correct." - The adaptive 64/256 KB byte limit is a real trade-off, not just a performance knob. It's what keeps typical requests fast, but it means content positioned late in a large page โ JSON-LD placed near the closing
</body>, a product block far down a long CMS-generated page โ can be missed purely because of where it sits in the HTML, independent of how "complex" the page seems. Larger/more script-heavy pages are paradoxically more likely to have relevant data pushed past the cutoff. Always checkquality.warningsfor aCONTENT_TRUNCATEDentry before treating a missing field as "this page doesn't have it" rather than "this API didn't read that far" โ there's no separate top-levelcontent_truncatedboolean, that signal lives insidequality. - Markdown conversion is not semantic understanding.
markdown_contentis a reasonably clean HTMLโMarkdown conversion, not a "read and understand what's actually the article" model โ it doesn't reliably distinguish primary content from navigation, related-content widgets, paywalled teasers, or boilerplate on every layout. - The cache trades freshness for speed, and that trade-off matters more for some fields than others. A 15-minute-stale
<title>is harmless; a 15-minute-stale price is not โ which is why product-bearing responses get a separate, much shorter TTL (PRODUCT_CACHE_TTL_SECONDS, 3 minutes by default) rather than sharing the general 15-minute cache. Everything else still follows the general TTL: don't build a real-time price-monitoring product directly on the general cache path without accounting for that window. - Arbitrary URL fetching is inherently unreliable, independent of this API: pages can be slow, redirect repeatedly, be enormous, block automated clients, have TLS quirks, serve different content by geography/User-Agent, be temporarily down, or return unexpected status codes. SSRF protections, timeouts, redirect limits, byte caps, and the circuit breaker (see above) bound the damage but can't make an unreliable target reliable.
- "~150โ300ms" describes the live-fetch network leg, not a ceiling on total response time. Server-side processing genuinely is single-digit milliseconds (see the benchmark methodology in
benchmarks/), but the live end-to-end time is dominated by however long the target site takes to respond โ if the target takes 2.5s, so does your request. Don't read the headline numbers as "this API responds in 200ms regardless of target." - There is no crawler. The unit of work is one URL in, one page's data out โ not "give me a domain and I'll discover/prioritize/crawl its pages for you."
/api/v1/linksclassifies up to 100 hyperlinks on the page you gave it; it doesn't follow them. Analyzing a whole site (homepage +/about+/pricing+/blog, deduplicated, prioritized, crawled) is something you'd orchestrate externally, one/extractcall per URL โ this project deliberately doesn't grow a queue/worker/crawl-frontier system into what's meant to stay a fast, stateless per-request API. /api/v1/contactsfinds public contact signals, not company or people intelligence. It returns emails/phones/social links it found in the page's HTML โ it does not identify who those belong to, their role, or verify anything. Turningwebsite -> public contact signalsinto real lead enrichment (domain -> company identification -> people -> roles -> verification -> confidence) is out of scope here; the endpoint is intentionally named/tagged "Contact Discovery," not "Lead Enrichment," to avoid implying more than it does.- Prefer the specific endpoint over
/api/v1/extractโ or usefields=./extractwithoutfieldsruns the complete pipeline./extract?fields=detected_technologiesnow runs exactly the same narrow extraction asGET /api/v1/tech-stack(and shares its cache entry) โfieldsgenuinely skips unneeded work rather than computing everything and filtering afterward. - This project's API surface is still evolving. New fields ship frequently (
quality,product_field_confidence,bot_protection_detected, and others were all added recently) โ additive changes only, existing fields never change type (see the versioning notes below), but if you're integrating deeply, treat the live/openapi.jsonschema as the source of truth over any single README snippet, which can lag by a commit or two.
None of this means the underlying approach is flawed for its intended job (fast metadata/SEO/contact/tech extraction from ordinary web pages) โ it means "extract everything from any URL via one HTTP request" has a ceiling that only browser automation (Playwright/Puppeteer) can raise, at the cost of the speed and low resource footprint that make this API fast in the first place.
โก Quick Start
Python
import requests
url = "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract"
headers = {
"X-RapidAPI-Key": "YOUR_RAPIDAPI_KEY_HERE",
"X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com"
}
params = {"url": "https://github.com"}
data = requests.get(url, headers=headers, params=params).json()
print(f"Title: {data['metadata']['title']}")
print(f"OG Type: {data['metadata']['og_type']}")
print(f"Robots: {data['metadata']['robots']}")
print(f"hreflang: {data['metadata']['hreflang_tags']}")
print(f"Product: {data['product_data']}")
print(f"Emails: {data['contacts']['emails']}")
print(f"Tech Stack: {data['detected_technologies']}")
print(f"Time: {data['execution_time_ms']} ms")
JavaScript / Node.js
const url = 'https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com';
const response = await fetch(url, {
headers: {
'X-RapidAPI-Key': 'YOUR_RAPIDAPI_KEY_HERE',
'X-RapidAPI-Host': 'web-metadata-and-contact-extractor.p.rapidapi.com'
}
});
const data = await response.json();
console.log('Title:', data.metadata.title);
console.log('OG Type:', data.metadata.og_type);
console.log('Product:', data.product_data);
console.log('Time:', data.execution_time_ms, 'ms');
cURL
curl --request GET \
--url 'https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com' \
--header 'X-RapidAPI-Host: web-metadata-and-contact-extractor.p.rapidapi.com' \
--header 'X-RapidAPI-Key: YOUR_RAPIDAPI_KEY_HERE'
๐ค MCP / AI Agent Support
This API is available as an MCP server out of the box, via RapidAPI's hosted gateway โ no extra setup or self-hosting needed. Add it to Claude Desktop, Claude Code, Cursor, or any MCP-compatible client:
{
"mcpServers": {
"RapidAPI Hub - Web Metadata and Contact Extractor": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.rapidapi.com",
"--header",
"x-api-host: web-metadata-and-contact-extractor.p.rapidapi.com",
"--header",
"x-api-key: YOUR_RAPIDAPI_KEY_HERE"
]
}
}
}
Once connected, an agent can call the extractor directly as a tool โ e.g. pulling markdown_content into a RAG pipeline, or checking detected_technologies and seo_score_percentage as part of an autonomous research/outreach workflow. Get a free key from the RapidAPI listing first.
๐ Sample API Response (JSON)
[!NOTE] This is real, live output from
GET /api/v1/extract?url=https://github.com, captured 2026-08-11 โ not a hand-edited illustration. It's deliberately left un-prettified, including the parts that aren't flattering:detected_technologiesincludes"Contentful"on a page that almost certainly isn't Contentful-powered (seetechnology_detailsbelow โ the evidence is one CDN hostname in anog:imageURL, hence only0.75confidence, not0.98), andmarkdown_content/word_countcome back empty because GitHub's homepage is a JS-heavy shell whose initial HTML has no<article>-like content for this API to find. Both are the "Honest Limitations" section below in action, not this example glossing over them.social_linkshas 20 keys total (see the Key Features table above); only non-null ones are worth reading, so it's trimmed here for length โ the real response includes all 20.
{
"url": "https://github.com",
"final_url": "https://github.com",
"status_code": 200,
"execution_time_ms": 130.43,
"bot_protection_detected": false,
"metadata": {
"title": "GitHub ยท Change is constant. GitHub keeps you ahead. ยท GitHub",
"description": "Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.",
"og_image": "https://images.ctfassets.net/8aevphvgewt8/4pe4eOtUJ0ARpZRE4fNekf/f52b1f9c52f059a33170229883731ed0/GH-Homepage-Universe-img.png",
"og_type": "object",
"canonical_url": "https://github.com",
"language": "en",
"favicon": "https://github.githubassets.com/favicons/favicon.svg",
"theme_color": "#1e2327",
"hreflang_tags": [
{"lang": "en-us", "url": "https://github.com"},
{"lang": "pt-br", "url": "https://github.com?locale=pt-br"}
],
"h1_tags": [],
"links_count": 51,
"content_length_bytes": 109075
},
"social_links": { "github": "https://github.com/features/copilot", "twitter": null, "linkedin": null, "...": "17 more platforms, all null here" },
"contacts": { "emails": [], "phones": [] },
"detected_technologies": ["Contentful"],
"technology_details": [
{ "name": "Contentful", "confidence": 0.75, "evidence": ["ctfassets.net"], "category": "cms" }
],
"product_data": null,
"quality": {
"score": 0.85,
"rendered": false,
"sources_used": ["meta"],
"warnings": [{ "field": null, "type": "CONTENT_TRUNCATED" }]
},
"rss_feeds": ["https://github.com/opensearch.xml", "https://github.githubassets.com/favicons/favicon.svg"],
"json_ld_schemas": [],
"security_score_percentage": 64.2,
"seo_score_percentage": 75.0,
"seo_passed_checks": ["Title tag present with optimal length (10-70 chars)", "Canonical link tag present", "OpenGraph image present for social sharing", "Favicon icon present"],
"seo_warnings": ["Meta description present but sub-optimal length (186 chars)", "Missing <h1> primary heading", "No structured data (JSON-LD) found"],
"internal_links": ["https://github.com/", "https://github.com/login", "https://github.com/features/copilot"],
"total_internal_count": 44,
"total_external_count": 5,
"word_count": 0,
"reading_time_minutes": 0.0,
"markdown_content": ""
}
This response is exactly
MetadataResponse(app/models/responses.py) serialized, trimmed for length โ every field shown above is real. Four model fields aren't shown:phone_detailsandproduct_field_confidence(both empty here โ no phone numbers orproduct_dataon this page),seo_checks(the structured, per-check version of theseo_passed_checks/seo_warningsalready shown above), andexternal_links(5 URLs, pertotal_external_count, omitted the same wayinternal_linksis truncated). There's no separatecontent_truncated/bytes_downloadedat the top level; that signal lives insidequality.warnings.
Since v4.0.0,
metadataalso includesviewport,twitter_card, andh1_count;/api/v1/tech-stackadditionally returnstechnology_details(confidence score, matched evidence, category per technology); andproduct_data(when present) includessku,mpn,gtin/isbn,seller,condition, and price-range fields. All additions are purely additive โ no existing field was removed or changed type. Fields that were previously untypedDict/List[Dict]blobs (metadata,technology_details,seo_checks,phone_details) now have documented Pydantic models in the OpenAPI schema โ withproduct_data, previously missing from/api/v1/extractdespite being shown here, now actually returned. Also new:product_field_confidenceand a top-levelqualityobject โ e.g. for a page where JSON-LD says โฌ39.99 and OpenGraph says โฌ29.99:"product_field_confidence": {"price": {"value": "39.99", "confidence": 0.5, "source": "json_ld", "agreement": ["json_ld", "microdata", "opengraph"]}}and"quality": {"score": 0.9, "rendered": false, "sources_used": ["json_ld", "microdata", "opengraph", "meta"], "warnings": [{"field": "product.price", "type": "SOURCE_CONFLICT", "values": {"json_ld": "39.99", "microdata": "39.99", "opengraph": "29.99"}, "chosen_source": "json_ld", "chosen_value": "39.99"}]}.
๐ API Endpoint Documentation
| Endpoint | Method | Description |
|---|---|---|
/api/v1/extract | GET | Full payload โ SEO, contacts, social, tech stack, schema, security, AI markdown, SEO audit, links, product data. Supports fields filter. |
/api/v1/link-preview | GET | Social link preview card โ title, description, OG image, favicon, site name, language. |
/api/v1/contacts | GET | Contact discovery โ public emails, phone numbers, social profiles found on the page (a raw signal for lead-gen workflows, not lead enrichment itself โ see "Honest Limitations"). |
/api/v1/tech-stack | GET | Framework & CMS detector โ 40+ technology signatures. |
/api/v1/schema | GET | Schema.org JSON-LD parser โ product prices, articles, events, organizations. |
/api/v1/security | GET | Security headers audit โ HSTS, CSP, X-Frame-Options, Referrer Policy with percentage score. |
/api/v1/markdown | GET | AI & LLM Markdown reader โ clean article text, word count, reading time. |
/api/v1/seo-audit | GET | Automated SEO diagnostic โ 14-point audit score with warnings list plus structured checks (severity/evidence per check). |
/api/v1/links | GET | Link classifier โ internal vs external hyperlinks (up to 100 per page). |
/health | GET | Health check โ status, version, protection mode. |
/health/details | GET | Operational health โ Redis mode/status, trust-proxy config. Requires X-Health-Secret if HEALTH_DETAILS_SECRET is set. |
/health/ready | GET | Readiness probe โ 200 once the HTTP client is initialized, 503 during startup. |
/metrics | GET | Prometheus scrape target โ request/cache/SSRF/circuit-breaker/rate-limit counters and latency histograms. Open by default (protect at the network/proxy level, standard Prometheus practice) โ set METRICS_SECRET to require an X-Metrics-Secret header instead, if you're self-hosting with the port reachable from the internet and no network-level control in front of it. Hidden from the public OpenAPI schema. |
Query Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
url | string | Yes | Target URL (e.g. https://example.com). Scheme-less inputs auto-normalized. |
fields | string | No | Comma-separated response filter (e.g. metadata,contacts). |
user_agent | string | No | Custom User-Agent header string. |
๐ง Self-Hosting & Local Development
# 1. Clone
git clone https://github.com/JosejuX/rapidapi-metadata-extractor.git
cd rapidapi-metadata-extractor/rapidapi_service
# 2. Install
pip install -r requirements.txt
# 3. Fixture-based test suite (SSRF matrix, circuit breaker, single-flight, rate-limit atomicity, ...)
pip install pytest
pytest tests/ -q
# 3b. Live-network smoke suite (14 SSRF vectors + 12 global domains)
python test_api.py
# 4. Load test (concurrent requests benchmark)
python load_test.py
# 5. Dev server
uvicorn main:app --reload --port 8000
Open http://localhost:8000/docs for the interactive Swagger UI.
Docker
# Pull the published image (built + pushed by .github/workflows/docker-publish.yml on every version tag)
docker pull ghcr.io/josejux/rapidapi-metadata-extractor:latest
docker run -p 8000:8000 ghcr.io/josejux/rapidapi-metadata-extractor:latest
# Or build it yourself
cd rapidapi_service
docker build -t rapidapi-metadata-extractor .
docker run -p 8000:8000 rapidapi-metadata-extractor
๐ท๏ธ Keywords
metadata-extractor โข opengraph-parser โข email-scraper โข contact-extractor โข social-links-finder โข tech-stack-detector โข seo-parser โข fastapi โข rapidapi โข python-web-scraper โข link-preview-generator โข lead-generation-api โข hreflang โข schema-org โข product-data-extractor โข ssrf-protection
๐ค Built With
This API was built and iterated on in collaboration with AI coding assistants โ Google's Gemini 3.6 Flash and Anthropic's Claude Sonnet 5 โ alongside human design, review, and testing decisions throughout.
๐ License
Distributed under the MIT License. See LICENSE for more information.