{ "results": [ { "url": "...", "format": "markdown", "content": "...", "markdown": "...", "title": "...", "method": "...", "cached": false, "fallbacks": ["jsonld"] }, { "url": "...", "format": "text", "content": "...", "title": "...", "method": "...", "cached": true } ] } ``` ### Structured Extraction API Extract structured fields from URL or raw HTML. ```bash curl -X POST https://md.genedai.me/api/extract \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "strategy": "css", "url": "https://example.com/article", "schema": { "fields": [ { "name": "title", "selector": "h1", "type": "text", "required": true }, { "name": "author", "selector": ".author", "type": "text" } ] }, "include_markdown": true }' ``` Batch extraction (`items`) is also supported (max 10 items). Additional extraction capabilities: - Use either top-level `url` / `html` or nested `input.url` / `input.html`. - `schema.fields[*].required` fails extraction when a required field is missing. - `options` supports `dedupe`, `includeEmpty`, and `regexFlags`. - `include_markdown: true` attaches converted markdown alongside extracted data. ### Job API (create / query / stream / run) Submit crawl/extract tasks as queued jobs, then run and monitor. Jobs are persisted as queued records in KV; execution begins when you call `/run`: ```bash # 1) Create job curl -X POST https://md.genedai.me/api/jobs \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -H "Idempotency-Key: demo-job-1" \ -d '{ "type": "crawl", "tasks": [ "https://example.com/a", "https://example.com/b" ], "priority": 10, "maxRetries": 2 }' # 2) Query status curl -H "Authorization: Bearer " \ https://md.genedai.me/api/jobs/ # 3) Watch status stream (SSE) curl -N -H "Authorization: Bearer " \ https://md.genedai.me/api/jobs//stream # 4) Execute queued tasks curl -X POST -H "Authorization: Bearer " \ https://md.genedai.me/api/jobs//run ``` Job API notes: - Supports both `type: "crawl"` and `type: "extract"`. - `type: "crawl"` accepts string URLs or object tasks with `format`, `selector`, `force_browser`, and `no_cache`. - `type: "extract"` reuses the same task shape as `/api/extract`. - `Idempotency-Key` is keyed by both the header value and request payload: same key + same payload returns the existing job; same key + different payload returns `409 Conflict`. - `priority` is normalized to `1..100` (default `10`), `maxRetries` to `0..10` (default `2`). - Up to `100` tasks are allowed per job. ### Deep Crawl API Run BFS/BestFirst deep crawl with filters/scoring and opt-in checkpoint resume. ```bash # non-stream curl -X POST https://md.genedai.me/api/deepcrawl \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "seed": "https://example.com/docs", "max_depth": 2, "max_pages": 20, "strategy": "best_first", "filters": { "allow_domains": ["example.com"], "url_patterns": ["https://example.com/docs/*"] }, "scorer": { "keywords": ["api", "reference"], "weight": 2 }, "checkpoint": { "crawl_id": "docs-crawl-001", "snapshot_interval": 5 } }' # stream mode (SSE: start/node/done/fail) curl -N -X POST https://md.genedai.me/api/deepcrawl \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "seed": "https://example.com/docs", "stream": true }' ``` Deep crawl request supports: - `include_external` to traverse off-domain links. - `filters.url_patterns`, `filters.allow_domains`, `filters.block_domains`, `filters.content_types`. - `scorer.keywords`, `scorer.weight`, `scorer.score_threshold`. - `output.include_markdown` to attach per-page markdown. - `fetch.selector`, `fetch.force_browser`, `fetch.no_cache` to control page conversion. - `checkpoint.crawl_id`, `checkpoint.resume`, `checkpoint.snapshot_interval`, `checkpoint.ttl_seconds`. ### Supported Sites Special adapters for optimal extraction on these platforms: | Site | Features | |---|---| | **WeChat** (`mp.weixin.qq.com`) | MicroMessenger UA, image proxy for hotlink bypass | | **Feishu/Lark Docs** (document surfaces such as `/wiki`, `/docx`, `/docs` on `.feishu.cn` / `.larksuite.com`) | Virtual scroll handling, R2 image storage, UI noise removal | | **Zhihu** (`zhihu.com/p/`) | Login wall removal, lazy image swap, hybrid proxy bypass | | **Yuque** (`yuque.com`) | SPA rendering, sidebar/toc removal | | **Notion** (`notion.site`, `notion.so`) | SPA rendering, lazy scroll loading | | **Juejin** (`juejin.cn/post/`) | Login popup removal, code block expansion | | **Twitter/X** (`twitter.com`, `x.com`) | Stealth rendering, login wall bypass | | **Reddit** (`reddit.com`) | URL transform to old.reddit.com, content extraction | | **CSDN** (`csdn.net`) | Login popup removal, code block expansion | | **36Kr** (`36kr.com`) | Stealth rendering, content extraction | | **Toutiao** (`toutiao.com`) | Stealth rendering, content extraction | | **NetEase** (`163.com`) | Content extraction | | **Weibo** (`weibo.com`) | Stealth rendering, hybrid proxy bypass | | **All other sites** | Generic mobile UA, lazy image handling | ### JavaScript / TypeScript ```ts const res = await fetch( "https://md.genedai.me/https://example.com/page?raw=true" ); const markdown = await res.text(); console.log(res.headers.get("X-Markdown-Method")); console.log(res.headers.get("X-Cache-Status")); // "HIT" or "MISS" ``` ### Python ```python import requests url = "https://md.genedai.me/https://example.com/page" resp = requests.get(url, params={"raw": "true", "format": "json"}) data = resp.json() print(data["title"], data["method"]) ``` ## API Endpoints | Endpoint | Method | Description | |---|---|---| | `/` | GET | Landing page with URL input form | | `/` | GET | Convert URL and render Markdown as HTML page | | `/?raw=true` | GET | Return raw Markdown as plain text | | `/?format=json` | GET | Return structured JSON (url, title, markdown, method) | | `/?format=html` | GET | Return HTML output for preview/basic rendering | | `/?format=text` | GET | Return plain text (no formatting) | | `/?selector=.class` | GET | Extract specific CSS selector | | `/?force_browser=true` | GET | Force browser rendering | | `/?engine=jina` | GET | Convert via Jina Reader API using the same output formats | | `/?engine=firecrawl` | GET | Convert via Firecrawl scrape using keyless mode when no key is configured | | `/?no_cache=true` | GET | Bypass KV cache | | `/api/stream?url=` | GET | SSE conversion stream (`step`, `done`, `fail`) with `selector` / `force_browser` / `no_cache` / `engine` / `token` support | | `/api/batch` | POST | Batch convert multiple URLs (max 10) | | `/api/extract` | POST | Structured extraction API (`css` / `xpath` / `regex`) | | `/api/jobs` | POST | Create queued crawl/extract job record | | `/api/jobs/:id` | GET | Query job status | | `/api/jobs/:id/stream` | GET | SSE job status stream | | `/api/jobs/:id/run` | POST | Execute queued/failed tasks in job | | `/api/deepcrawl` | POST | Deep crawl API (BFS/BestFirst, stream/non-stream, checkpoint) | | `/api/og` | GET | Dynamic Open Graph image for landing/rendered pages | | `/img/` | GET | Image proxy (bypasses hotlink protection) | | `/r2img/` | GET | Serve image from R2 storage | | `/api/health` | GET | Health + runtime + operational metrics | ## Authentication Matrix The hosted instance at `md.genedai.me` uses D1-backed API keys with tiers (see [API Keys and Tiers](#api-keys-and-tiers)). Self-hosted deployments can skip the `AUTH_DB` binding and fall back to the legacy `API_TOKEN` / `PUBLIC_API_TOKEN` secrets. | Route Group | Anonymous | Free tier (`mk_…`) | Pro tier (`mk_…`) | |---|---|---|---| | `GET /` | ✅ cache + readability + keyless `engine=jina/firecrawl` | ✅ full pipeline + keyless `engine=jina/firecrawl` | ✅ + all engines, `no_cache`, `force_browser` | | `GET /api/stream` | ✅ cache + readability + keyless `engine=jina/firecrawl` | ✅ full pipeline + keyless `engine=jina/firecrawl` | ✅ full + params | | `POST /api/batch` | ❌ 401 | ✅ | ✅ | | `POST /api/extract` | ❌ 401 | ✅ | ✅ | | `POST /api/deepcrawl` | ❌ 401 | ✅ | ✅ | | `POST /api/jobs*` | ❌ 401 | ✅ | ✅ | | `GET /api/me`, `/api/keys`, `/api/usage` | — | session cookie | session cookie or Bearer key | | `POST /api/auth/magic-link`, `/auth/logout` | public | public | public | | `GET /api/auth/verify` | public (single-use token) | — | — | | `GET /portal/` (SPA) | public HTML | — | — | | `GET /api/health`, `/llms.txt`, `/robots.txt`, `/sitemap.xml` | public | public | public | The batch / extract / deepcrawl / jobs endpoints are always gated because they either fan out into many conversions or touch Browser Rendering directly. ## Response Headers (Raw API) | Header | Description | |---|---| | `Content-Type` | `text/markdown`, `application/json`, `text/html`, or `text/plain` | | `X-Source-URL` | The original target URL | | `X-Markdown-Tokens` | Token count (native Markdown for Agents only) | | `X-Markdown-Native` | `"true"` when native, `"false"` otherwise | | `X-Markdown-Method` | `"native"`, `"readability+turndown"`, `"browser+readability+turndown"`, `"jina"`, `"firecrawl"`, or `"cf"` | | `X-Cache-Status` | `"HIT"` or `"MISS"` | | `X-Markdown-Fallbacks` | Comma-separated fallback list (when used) | | `X-Browser-Rendered` | `"true"` when browser rendering path was used | | `X-Paywall-Detected` | `"true"` when paywall heuristics were triggered | | `X-RateLimit-Limit` | Monthly credit quota (authenticated requests only) | | `X-RateLimit-Remaining` | Credits remaining this month | | `X-Request-Cost` | Fixed per-request-type credit cost | | `X-Quota-Exceeded` | `"true"` when quota is exhausted but a cached response was served | | `Retry-After` | Present on `429` responses (IP rate limit or quota exceeded) | | `Access-Control-Allow-Origin` | `*` — CORS enabled | ## Features | Feature | Description | |---|---| | **Any Website** | Works on every site with four conversion paths | | **Site Adapters** | Specialized extractors for WeChat, Feishu, Zhihu, Yuque, Notion, Juejin | | **Anti-Bot Bypass** | Browser Rendering handles JS challenges, CAPTCHAs, and verification | | **3-Tier Cache** | In-memory hot cache → Cloudflare Cache API (per-colo, free) → KV (global, persistent) | | **Developer Portal** | Self-service signup, API key management, real-time usage dashboard | | **Tier System** | Anonymous (cache+readability only), Free (1k/mo), Pro (50k/mo) | | **R2 Image Storage** | Images stored reliably, served via proxy URLs | | **Multiple Formats** | Markdown, HTML, text, or structured JSON output | | **CSS Selectors** | Target specific page elements for extraction | | **Batch API v2** | Convert up to 10 URLs with per-item format/selector/browser/cache options | | **Structured Extraction** | CSS/XPath/Regex extraction via `/api/extract` with optional markdown attachment | | **Job Dispatcher** | Queue + run + monitor crawl/extract workloads via `/api/jobs/*` | | **Deep Crawl** | BFS + BestFirst traversal, filters/scorers, stream mode, checkpoint/resume | | **Table Support** | Improved handling of simple and complex tables | | **Smart Extraction** | Readability strips nav, ads, sidebars — extracts main article content | | **Rendered View** | Dark-themed Markdown preview with GitHub CSS and tab switching | | **Session Profiles** | Persist/replay cookies and localStorage for repeat authenticated crawling | | **Proxy Pool Fallback** | Multi-proxy + UA/header variant rotation for challenge-prone targets | | **SSRF Protection** | Blocks private IPs, IPv6 link-local, cloud metadata endpoints | | **Timeout Protection** | Time-budgeted scrolling for Feishu virtual scroll documents | | **Built-in Rate Limiting** | Per-IP limits for conversion, stream, and batch routes | | **Runtime Paywall Rules** | Support dynamic paywall rule updates via env/KV JSON | | **Operational Health** | `/api/health` exposes throughput/success/retry/backlog and P50/P95 latency | ## Tech Stack | Component | Role | |---|---| | [Cloudflare Workers](https://workers.cloudflare.com/) | Edge runtime — global deployment | | [Cloudflare Browser Rendering](https://developers.cloudflare.com/browser-rendering/) | Headless Chrome for JS-heavy/anti-bot pages | | [Cloudflare KV](https://developers.cloudflare.com/kv/) | Edge key-value cache for converted content | | [Cloudflare R2](https://developers.cloudflare.com/r2/) | Object storage for images | | [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) | Native HTML→Markdown at edge | | [@mozilla/readability](https://github.com/mozilla/readability) | Article content extraction (Firefox Reader View) | | [Turndown](https://github.com/mixmark-io/turndown) | HTML→Markdown conversion | | [@cloudflare/puppeteer](https://github.com/nichochar/puppeteer) | Puppeteer API for Browser Rendering | | [LinkeDOM](https://github.com/WebReflection/linkedom) | Lightweight DOM for Workers | | [Vitest](https://vitest.dev/) | Unit testing framework | ## AI Agent Integration Three ways to use Website2Markdown from AI agents: ### Agent Skills (Claude Code, OpenClaw, Claw) One command install, auto-discovered by your agent. Includes usage patterns, error handling, and guides for all 21 adapters. ```bash # Claude Code git clone https://github.com/Digidai/website2markdown-skills ~/.claude/skills/website2markdown # Codex CLI git clone https://github.com/Digidai/website2markdown-skills ~/.codex/skills/website2markdown # Gemini CLI git clone https://github.com/Digidai/website2markdown-skills ~/.gemini/skills/website2markdown # OpenClaw npx clawhub@latest install website2markdown ``` One command, auto-discovered in new sessions. See the [website2markdown-skills](https://github.com/Digidai/website2markdown-skills) repo for full documentation. ### MCP Server (Claude Desktop, Cursor IDE, Windsurf) Standard MCP protocol with `convert_url` tool. ```bash npm install -g @digidai/mcp-website2markdown ``` Claude Desktop config (`~/.claude/claude_desktop_config.json`): ```json { "mcpServers": { "website2markdown": { "command": "mcp-website2markdown", "env": { "WEBSITE2MARKDOWN_API_URL": "https://md.genedai.me" } } } } ```

@digidai/mcp-website2markdown

Installation