Bỏ qua tới nội dung
TakoAPI
Quay lại kỹ năng

midscene-computer-automation

bởi quanru

Web & Frontend Development
1.4kLượt tải do clawskills.sh công bố
2Sao
0Lượt xem
0 lượt thíchĐăng nhập để thích kỹ năng này
—Chưa có đánh giá
Đánh giá của bạn

Tóm tắt

Vision-driven desktop automation using Midscene.

Mô tả

Kỹ năng này làm gì

Controls macOS, Windows, and Linux desktops with natural language using Midscene.js. Operates from screenshots, so it works with any visible UI without requiring DOM access or accessibility labels. Automates clicks, typing, window management, and app interactions across any application.

Unlike Selenium or AppleScript, it works with any app on any stack because it reads the screen directly rather than relying on DOM structure or accessibility APIs.

Khi nào nên dùng

  • Automating repetitive form entry in a legacy desktop app
  • Taking targeted screenshots of specific windows for documentation
  • Launching apps and navigating their menus via text commands
  • Extracting visible on-screen data from any GUI application
  • Running multi-step workflows across different desktop programs

Quy trình ví dụ

ĐẦU VÀO

User asks: Open Chrome, search for the weather in New York, and tell me the result

AGENT
  1. 1

    Connect to the desktop with npx @midscene/computer@1 connect

  2. 2

    Open Chrome using the system launcher (open -a 'Google Chrome' on macOS)

  3. 3

    Take a screenshot to confirm Chrome is in the foreground

  4. 4

    Run act --prompt "search for weather in New York and read the current conditions from the page" to perform the search and extract results

  5. 5

    Disconnect and report the weather information extracted from the screen

ĐẦU RA

Current weather conditions for New York, with a screenshot showing the search result page

Yêu cầu

  • MIDSCENE_MODEL_API_KEY from a supported visual AI model provider (Google Gemini, Alibaba Dashscope, Doubao, or OpenRouter)
  • MIDSCENE_MODEL_NAME, MIDSCENE_MODEL_BASE_URL, and MIDSCENE_MODEL_FAMILY set to match the chosen provider

Cài đặt

clawhub install quanru/midscene-computer-automation

Chạy lệnh CLI, hoặc dán prompt vào Claude Code, Cursor, Codex hay coding agent bạn đang dùng.

Liên kết