본문으로 건너뛰기
TakoAPI
스킬로 돌아가기

midscene-computer-automation

quanru 제작

Web & Frontend Development
1.4kclawskills.sh가 보고한 다운로드 수
2스타
0조회수
좋아요 0개로그인하면 이 스킬에 좋아요를 누를 수 있습니다
—아직 평가 없음
내 평가

요약

Vision-driven desktop automation using Midscene.

설명

이 스킬이 하는 일

Controls macOS, Windows, and Linux desktops with natural language using Midscene.js. Operates from screenshots, so it works with any visible UI without requiring DOM access or accessibility labels. Automates clicks, typing, window management, and app interactions across any application.

Unlike Selenium or AppleScript, it works with any app on any stack because it reads the screen directly rather than relying on DOM structure or accessibility APIs.

사용 시점

  • Automating repetitive form entry in a legacy desktop app
  • Taking targeted screenshots of specific windows for documentation
  • Launching apps and navigating their menus via text commands
  • Extracting visible on-screen data from any GUI application
  • Running multi-step workflows across different desktop programs

예시 워크플로

입력

User asks: Open Chrome, search for the weather in New York, and tell me the result

에이전트
  1. 1

    Connect to the desktop with npx @midscene/computer@1 connect

  2. 2

    Open Chrome using the system launcher (open -a 'Google Chrome' on macOS)

  3. 3

    Take a screenshot to confirm Chrome is in the foreground

  4. 4

    Run act --prompt "search for weather in New York and read the current conditions from the page" to perform the search and extract results

  5. 5

    Disconnect and report the weather information extracted from the screen

출력

Current weather conditions for New York, with a screenshot showing the search result page

요구 사항

  • MIDSCENE_MODEL_API_KEY from a supported visual AI model provider (Google Gemini, Alibaba Dashscope, Doubao, or OpenRouter)
  • MIDSCENE_MODEL_NAME, MIDSCENE_MODEL_BASE_URL, and MIDSCENE_MODEL_FAMILY set to match the chosen provider

설치

clawhub install quanru/midscene-computer-automation

CLI 명령을 실행하거나, 프롬프트를 Claude Code, Cursor, Codex 등 사용하는 코딩 에이전트에 붙여넣으세요.

링크