crawl-for-ai
bởi angusthefuzz
Web & Frontend DevelopmentTóm tắt
Web scraping using local Crawl4AI instance.
Mô tả
Kỹ năng này làm gì
Fetches full web page content using a local Crawl4AI instance with JavaScript rendering. Offers two endpoints: a proxy for clean markdown output and a direct endpoint for structured data including links, HTML, and media. No usage limits since it runs locally.
Runs on a local instance with no rate limits or API costs, and reliably handles dynamic JavaScript content that simpler fetch-based scrapers miss.
Khi nào nên dùng
- Scraping JavaScript-rendered single-page applications
- Extracting article text from JS-heavy or dynamic news sites
- Collecting product data from e-commerce pages
- Archiving web pages with full link and media metadata
- Pulling content from documentation portals that require JS execution
Quy trình ví dụ
User asks: Get the full content from https://example.com/docs/getting-started
- 1
Identify the target URL from the user request
- 2
Call crawl4ai.js with the URL via the proxy endpoint for clean output
- 3
Receive the page rendered as structured markdown with metadata
- 4
Return the extracted content to the user
Full page content as clean markdown, including any dynamically loaded text
Yêu cầu
- Self-hosted Crawl4AI instance running locally or on a server
- CRAWL4AI_URL environment variable set to the instance URL
- CRAWL4AI_KEY environment variable if the instance requires authentication
Cài đặt
clawhub install angusthefuzz/crawl-for-aiChạy lệnh CLI, hoặc dán prompt vào Claude Code, Cursor, Codex hay coding agent bạn đang dùng.