Zum Inhalt springen
TakoAPI
Zurück zu den Skills

crawl-for-ai

von angusthefuzz

Web & Frontend Development
2.4kDownloadzahlen laut clawskills.sh
5Sterne
0Aufrufe
0 LikesMelde dich an, um diesen Skill zu liken
—Noch nicht bewertet
Deine Bewertung

Kurzbeschreibung

Web scraping using local Crawl4AI instance.

Beschreibung

Was dieser Skill macht

Fetches full web page content using a local Crawl4AI instance with JavaScript rendering. Offers two endpoints: a proxy for clean markdown output and a direct endpoint for structured data including links, HTML, and media. No usage limits since it runs locally.

Runs on a local instance with no rate limits or API costs, and reliably handles dynamic JavaScript content that simpler fetch-based scrapers miss.

Wann du ihn einsetzt

  • Scraping JavaScript-rendered single-page applications
  • Extracting article text from JS-heavy or dynamic news sites
  • Collecting product data from e-commerce pages
  • Archiving web pages with full link and media metadata
  • Pulling content from documentation portals that require JS execution

Beispiel-Workflow

EINGABE

User asks: Get the full content from https://example.com/docs/getting-started

AGENT
  1. 1

    Identify the target URL from the user request

  2. 2

    Call crawl4ai.js with the URL via the proxy endpoint for clean output

  3. 3

    Receive the page rendered as structured markdown with metadata

  4. 4

    Return the extracted content to the user

AUSGABE

Full page content as clean markdown, including any dynamically loaded text

Voraussetzungen

  • Self-hosted Crawl4AI instance running locally or on a server
  • CRAWL4AI_URL environment variable set to the instance URL
  • CRAWL4AI_KEY environment variable if the instance requires authentication

Installation

clawhub install angusthefuzz/crawl-for-ai

Führe den CLI-Befehl aus oder füge den Prompt in Claude Code, Cursor, Codex oder deinen Coding-Agenten ein.

Links