跳到正文
TakoAPI
返回技能列表

senior-data-engineer

由 alirezarezvani 提供

Data & Analytics
1.8k下载量由 clawskills.sh 提供
2星标
1浏览量
0 个赞登录后即可为这个技能点赞
—暂无评分
你的评分

简介

Data engineering skill for building scalable.

描述

该技能的功能

Covers production-grade data pipeline design, ETL/ELT implementation, and data infrastructure decisions. Handles batch and streaming architectures using Python, SQL, Spark, Airflow, dbt, and Kafka. Includes data modeling, quality validation, and DataOps practices.

Combines architecture decision frameworks with ready-to-run scripts for orchestration, quality validation, and performance analysis, covering the full modern data stack in one place.

何时使用

  • Building a daily batch ETL from Postgres to Snowflake
  • Setting up Airflow DAGs for scheduled data ingestion
  • Validating data quality on freshness, completeness, and uniqueness
  • Choosing between Lambda and Kappa pipeline architectures
  • Optimizing a slow Spark job processing large parquet files

示例工作流

输入

User asks: Design a pipeline to ingest sales data from Postgres into Snowflake daily

智能体
  1. 1

    Evaluates latency and volume requirements to confirm batch processing is appropriate

  2. 2

    Generates an Airflow DAG config with a scheduled daily run targeting Snowflake

  3. 3

    Defines dbt models for transformation and dimensional modeling in the warehouse

  4. 4

    Adds data quality checks for freshness, completeness, and uniqueness on the sales dataset

输出

Airflow DAG, dbt model files, and a data quality validation script ready for deployment

安装

clawhub install alirezarezvani/senior-data-engineer

运行 CLI 命令,或把这段提示词粘贴到 Claude Code、Cursor、Codex 或你在用的任意编程智能体里。

链接