homelab-cluster
bởi mlesnews
DevOps & CloudTóm tắt
Manage multi-tier AI inference clusters for homelabs.
Mô tả
Kỹ năng này làm gì
Manages multi-tier AI inference clusters for homelab environments. Covers health monitoring across GPU and CPU nodes, MoE-based request routing, automatic recovery for downed nodes, and model deployment via Ollama and llama.cpp. Includes GPU memory planning, Docker volume strategies, and a unified LiteLLM API gateway.
Consolidates health checks, failover routing, and node recovery into one workflow instead of managing each Ollama or llama.cpp instance separately.
Khi nào nên dùng
- Routing coding tasks to a dedicated coder model on the local GPU node
- Recovering a remote Docker Desktop crash without manual intervention
- Planning which quantized models fit within 24GB VRAM across a cluster
- Restarting GPU containers sequentially to avoid CUDA initialization deadlocks
- Exposing all cluster models through a single OpenAI-compatible endpoint
Quy trình ví dụ
User asks: route this code review request to the best available model
- 1
Classify the prompt as a 'code' task
- 2
Query the health endpoint for the designated coder model node
- 3
If healthy, return the coder model endpoint; if degraded, check fallback tier
- 4
Select next-best available model from health results
- 5
Return the chosen model endpoint and tier metadata to the caller
Request routed to remote/mark-i running Qwen2.5-Coder-7B at full GPU capacity
Yêu cầu
- SSH access credentials stored in a vault (e.g., Azure Key Vault or HashiCorp Vault) for remote node recovery
Cài đặt
clawhub install mlesnews/homelab-clusterChạy lệnh CLI, hoặc dán prompt vào Claude Code, Cursor, Codex hay coding agent bạn đang dùng.