تخطَّ إلى المحتوى
TakoAPI
العودة إلى المهارات

web-scraper

بواسطة guifav

Web & Frontend Development
1.7kالتنزيلات وفق ما يذكره clawskills.sh
0المشاهدات
لا إعجاباتسجّل الدخول للإعجاب بهذه المهارة
—لم يُقيَّم بعد
تقييمك

موجز

Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection.

الوصف

ما الذي تفعله هذه المهارة

A 5-stage web scraping pipeline that detects news articles, extracts content using a cascade of strategies (static HTTP, Playwright, Scrapy), removes boilerplate, and pulls structured metadata. Handles JS-rendered pages, soft paywalls, bulk crawls, and encoding normalization. Optionally extracts named entities via LLM using clean text only, never raw HTML.

The cascade fallback (static HTTP first, Playwright only when needed) avoids headless browser overhead for simple pages while still handling JS-rendered SPAs that single-strategy scrapers fail on.

متى تستخدمها

  • Extracting full article text and metadata from a news site
  • Bulk crawling all articles on a domain into structured JSON
  • Scraping content from a React or Next.js SPA that requires JS rendering
  • Extracting people, organizations, and events mentioned in news articles
  • Checking which parts of a paywalled article are publicly accessible

مثال على سير العمل

الإدخال

User asks: Scrape this news article URL and extract all named entities

الوكيل
  1. 1

    Analyzes URL pattern and page HTML to confirm it is a news article (Stage 1)

  2. 2

    Fetches page with static HTTP; detects JS rendering need and escalates to Playwright (Stage 2)

  3. 3

    Removes boilerplate with trafilatura and normalizes encoding (Stage 3)

  4. 4

    Extracts structured metadata: title, author, publication date, paywall status (Stage 4)

  5. 5

    Sends clean text to LLM via OpenRouter and parses entity JSON response (Stage 5)

الإخراج

JSON file with article text, metadata fields, and structured entities grouped by people, organizations, locations, and events

المتطلبات

  • OPENROUTER_API_KEY environment variable (for LLM entity extraction in Stage 5)
  • Playwright Chromium browser installed via 'playwright install chromium'

التثبيت

clawhub install guifav/web-scraper

شغّل أمر واجهة سطر الأوامر، أو الصق النص في Claude Code أو Cursor أو Codex أو أي وكيل برمجة تستخدمه.

الروابط