Skip to content
TakoAPI
Back to skills

search-cluster

by 1999azzar

Search & Research
879Downloads reported by clawskills.sh
0Views
0 likesSign in to like this skill
—Not yet rated
Your rating

Brief

Unified search aggregator for Google, Wikipedia, Reddit, NewsAPI, and RSS feeds with optional Redis caching.

Description

What This Skill Does

Multi-provider search aggregator that queries Google CSE, Wikipedia, Reddit, GNews RSS, and DuckDuckGo (via headless scraping) in a single call. Returns structured JSON with source, title, link, and sanitized snippet per result. Supports optional Redis caching and runs the scraping provider in a subprocess-isolated virtual environment.

Querying five independent providers in one call reduces coverage gaps that occur when any single source is rate-limited, blocked, or returns poor results.

When to Use It

  • Aggregating news and forum discussion results on a single topic across five sources
  • Cross-referencing Wikipedia summaries with Reddit threads during research
  • Building a search backend with automatic provider-level fallback
  • Fetching DuckDuckGo results without triggering bot detection
  • Caching repeated search queries to reduce API call volume

Example Workflow

INPUT

User asks: "Find recent news and discussions about LLM fine-tuning"

AGENT
  1. 1

    Invokes scripts/search-cluster.py with provider flag 'all' and the query string

  2. 2

    Dispatches requests to Google CSE, Wikipedia, Reddit, and GNews RSS concurrently

  3. 3

    Spawns an isolated subprocess running stealth_fetch.py to query DuckDuckGo via Scrapling

  4. 4

    Sanitizes and normalizes all provider responses into a shared schema

  5. 5

    Writes aggregated results to Redis cache if REDIS_HOST is configured

OUTPUT

Structured JSON array where each entry contains source, title, link, and sanitized snippet

Requirements

  • GOOGLE_API_KEY (optional, Google Custom Search)
  • GOOGLE_CSE_ID (optional, Google Custom Search Engine ID)
  • SCRAPLING_PYTHON_PATH (optional, path to scrapling venv python binary)
  • REDIS_HOST and REDIS_PORT (optional, for result caching)

Installation

clawhub install 1999azzar/search-cluster

Run the CLI command, or paste the prompt into Claude Code, Cursor, Codex or whichever coding agent you use.

Links