bigquery-pipeline-audit

Overview

BigQuery Pipeline Audit is a skill that performs comprehensive auditing of Python + BigQuery data pipelines for cost safety, idempotency, and production readiness. It analyzes every BigQuery job trigger and external call in the codebase, identifies cost exposure risks, verifies dry-run capabilities, and produces a structured report with exact function names, line locations, and minimal fix suggestions.

Key Features

• Cost Exposure Analysis - Locates every client.query, load_table_from_*, extract_table, and copy_table call, evaluating worst-case billing scenarios, loop multipliers, and missing maximum_bytes_billed safeguards
• Dry Run & Execution Mode Verification - Checks for a --mode flag with dry_run and execute options, ensuring dry runs produce zero billed execution and production requires explicit confirmation
• Idempotency Checking - Verifies that pipeline reruns do not corrupt data by checking for proper backfill patterns, write disposition settings, and deduplication mechanisms
• Structured Audit Report - Produces a categorized report (A through F plus Final) with exact function names, line references, and minimal patch suggestions rather than full rewrites
• External Call Tracking - Identifies all API calls, LLM invocations, and storage writes alongside BigQuery operations to give a complete cost picture

Who is this for?

This skill is for data engineers and platform teams managing Python + BigQuery pipelines who need to ensure cost safety and production readiness before deployment. It is especially valuable for teams processing large datasets where unchecked queries in loops or missing billing limits could result in significant unexpected costs.