Bỏ qua tới nội dung
TakoAPI
Quay lại kỹ năng

senior-data-engineer

bởi alirezarezvani

Data & Analytics
1.8kLượt tải do clawskills.sh công bố
2Sao
1Lượt xem
0 lượt thíchĐăng nhập để thích kỹ năng này
—Chưa có đánh giá
Đánh giá của bạn

Tóm tắt

Data engineering skill for building scalable.

Mô tả

Kỹ năng này làm gì

Covers production-grade data pipeline design, ETL/ELT implementation, and data infrastructure decisions. Handles batch and streaming architectures using Python, SQL, Spark, Airflow, dbt, and Kafka. Includes data modeling, quality validation, and DataOps practices.

Combines architecture decision frameworks with ready-to-run scripts for orchestration, quality validation, and performance analysis, covering the full modern data stack in one place.

Khi nào nên dùng

  • Building a daily batch ETL from Postgres to Snowflake
  • Setting up Airflow DAGs for scheduled data ingestion
  • Validating data quality on freshness, completeness, and uniqueness
  • Choosing between Lambda and Kappa pipeline architectures
  • Optimizing a slow Spark job processing large parquet files

Quy trình ví dụ

ĐẦU VÀO

User asks: Design a pipeline to ingest sales data from Postgres into Snowflake daily

AGENT
  1. 1

    Evaluates latency and volume requirements to confirm batch processing is appropriate

  2. 2

    Generates an Airflow DAG config with a scheduled daily run targeting Snowflake

  3. 3

    Defines dbt models for transformation and dimensional modeling in the warehouse

  4. 4

    Adds data quality checks for freshness, completeness, and uniqueness on the sales dataset

ĐẦU RA

Airflow DAG, dbt model files, and a data quality validation script ready for deployment

Cài đặt

clawhub install alirezarezvani/senior-data-engineer

Chạy lệnh CLI, hoặc dán prompt vào Claude Code, Cursor, Codex hay coding agent bạn đang dùng.

Liên kết