Bỏ qua tới nội dung
TakoAPI
Quay lại kỹ năng

crawl-for-ai

bởi angusthefuzz

Web & Frontend Development
2.4kLượt tải do clawskills.sh công bố
5Sao
0Lượt xem
0 lượt thíchĐăng nhập để thích kỹ năng này
—Chưa có đánh giá
Đánh giá của bạn

Tóm tắt

Web scraping using local Crawl4AI instance.

Mô tả

Kỹ năng này làm gì

Fetches full web page content using a local Crawl4AI instance with JavaScript rendering. Offers two endpoints: a proxy for clean markdown output and a direct endpoint for structured data including links, HTML, and media. No usage limits since it runs locally.

Runs on a local instance with no rate limits or API costs, and reliably handles dynamic JavaScript content that simpler fetch-based scrapers miss.

Khi nào nên dùng

  • Scraping JavaScript-rendered single-page applications
  • Extracting article text from JS-heavy or dynamic news sites
  • Collecting product data from e-commerce pages
  • Archiving web pages with full link and media metadata
  • Pulling content from documentation portals that require JS execution

Quy trình ví dụ

ĐẦU VÀO

User asks: Get the full content from https://example.com/docs/getting-started

AGENT
  1. 1

    Identify the target URL from the user request

  2. 2

    Call crawl4ai.js with the URL via the proxy endpoint for clean output

  3. 3

    Receive the page rendered as structured markdown with metadata

  4. 4

    Return the extracted content to the user

ĐẦU RA

Full page content as clean markdown, including any dynamically loaded text

Yêu cầu

  • Self-hosted Crawl4AI instance running locally or on a server
  • CRAWL4AI_URL environment variable set to the instance URL
  • CRAWL4AI_KEY environment variable if the instance requires authentication

Cài đặt

clawhub install angusthefuzz/crawl-for-ai

Chạy lệnh CLI, hoặc dán prompt vào Claude Code, Cursor, Codex hay coding agent bạn đang dùng.

Liên kết