search-cluster
1999azzar द्वारा
Search & Researchसंक्षिप्त
Unified search aggregator for Google, Wikipedia, Reddit, NewsAPI, and RSS feeds with optional Redis caching.
विवरण
यह स्किल क्या करती है
Multi-provider search aggregator that queries Google CSE, Wikipedia, Reddit, GNews RSS, and DuckDuckGo (via headless scraping) in a single call. Returns structured JSON with source, title, link, and sanitized snippet per result. Supports optional Redis caching and runs the scraping provider in a subprocess-isolated virtual environment.
Querying five independent providers in one call reduces coverage gaps that occur when any single source is rate-limited, blocked, or returns poor results.
इसे कब इस्तेमाल करें
- Aggregating news and forum discussion results on a single topic across five sources
- Cross-referencing Wikipedia summaries with Reddit threads during research
- Building a search backend with automatic provider-level fallback
- Fetching DuckDuckGo results without triggering bot detection
- Caching repeated search queries to reduce API call volume
उदाहरण वर्कफ़्लो
User asks: "Find recent news and discussions about LLM fine-tuning"
- 1
Invokes scripts/search-cluster.py with provider flag 'all' and the query string
- 2
Dispatches requests to Google CSE, Wikipedia, Reddit, and GNews RSS concurrently
- 3
Spawns an isolated subprocess running stealth_fetch.py to query DuckDuckGo via Scrapling
- 4
Sanitizes and normalizes all provider responses into a shared schema
- 5
Writes aggregated results to Redis cache if REDIS_HOST is configured
Structured JSON array where each entry contains source, title, link, and sanitized snippet
आवश्यकताएँ
- GOOGLE_API_KEY (optional, Google Custom Search)
- GOOGLE_CSE_ID (optional, Google Custom Search Engine ID)
- SCRAPLING_PYTHON_PATH (optional, path to scrapling venv python binary)
- REDIS_HOST and REDIS_PORT (optional, for result caching)
इंस्टॉलेशन
clawhub install 1999azzar/search-clusterCLI कमांड चलाएँ, या प्रॉम्प्ट को Claude Code, Cursor, Codex या जो भी कोडिंग एजेंट आप इस्तेमाल करते हैं उसमें पेस्ट करें।