web-fetch
🎯Skillfrom 0xbigboss/claude-code
|Fetches web content and converts HTML to clean markdown using intelligent content extraction and CSS selectors.
Installation
npx skills add https://github.com/0xbigboss/claude-code --skill web-fetchSkill Details
Fetches web content with intelligent content extraction, converting HTML to clean markdown. Use for documentation, articles, and reference pages http/https URLs.
Overview
# Web Content Fetching
Fetch web content using curl | html2markdown with CSS selectors for clean, complete markdown output.
Quick Usage (Known Sites)
Use site-specific selectors for best results:
```bash
# Anthropic docs
curl -s "
# MDN Web Docs
curl -s "
# GitHub docs
curl -s "
# Generic article pages
curl -s "
```
Site Patterns
| Site | Include Selector | Exclude Selector |
|------|------------------|------------------|
| platform.claude.com | #content-container | - |
| docs.anthropic.com | #content-container | - |
| developer.mozilla.org | article | - |
| github.com (docs) | article | nav,.sidebar |
| Generic | article,main | nav,header,footer,script,style |
Universal Fallback (Unknown Sites)
For sites without known patterns, use the Bun script which auto-detects content:
```bash
bun ~/.claude/skills/web-fetch/fetch.ts "
```
Setup (one-time)
```bash
cd ~/.claude/skills/web-fetch && bun install
```
Finding the Right Selector
When a site isn't in the patterns list:
```bash
# Check what content containers exist
curl -s "
# Test a selector
curl -s "
# Check line count
curl -s "
```
Options Reference
```bash
--include-selector "CSS" # Only include matching elements
--exclude-selector "CSS" # Remove matching elements
--domain "https://..." # Convert relative links to absolute
```
Comparison
| Method | Anthropic Docs | Code Blocks | Complexity |
|--------|----------------|-------------|------------|
| Full page | 602 lines | Yes | Noisy |
| --include-selector "#content-container" | 385 lines | Yes | Clean |
| Bun script (universal) | 383 lines | Yes | Clean |
Troubleshooting
Wrong content selected: The site may have multiple articles. Inspect the HTML:
```bash
curl -s "
```
Empty output: The selector doesn't match. Try broader selectors like main or body.
Missing code blocks: Check if the site uses non-standard code formatting.
Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.
More from this repository10
Guides React developers in writing clean, efficient components by providing best practices for hooks, effects, and component design.
Guides Python developers in implementing type-first development with robust type checking, dataclasses, discriminated unions, and domain-specific type primitives.
Generates images using OpenAI's image generation API within the Claude Code environment.
Optimizes Tamagui configurations and styling patterns for high-performance, cross-platform React Native and web applications with v4 compiler settings.
Provides Playwright test best practices for creating resilient, maintainable tests using user-facing locators, Page Object Models, and robust testing patterns.
Enforces type-first Go development with custom types, interfaces, and patterns to create robust, self-documenting code with strong compile-time guarantees.
Enforces type-first development in Zig by leveraging tagged unions, explicit error sets, comptime validation, and type safety techniques.
Enforces TypeScript best practices by guiding developers to create robust, type-safe code through compile-time validation, discriminated unions, and explicit type definitions.
Generates environment configuration files or environment variables for development projects, ensuring consistent and reproducible setup across different environments.
Retrieves Zig language documentation from Codeberg sources, standard library files, and online references via CLI commands.