datanalysis-credit-risk

Overview

A GitHub Copilot skill providing a complete data cleaning and variable screening pipeline for credit risk pre-loan modeling. It executes an 11-step process from raw data loading through organization-level analysis, abnormal period filtering, missing rate calculation, IV/PSI-based variable selection, null importance denoising, high-correlation removal, and cleaning report generation — all without modifying original data.

Key Features

• 11-step automated pipeline: Executes data loading, organization sample analysis, OOS data separation, abnormal month filtering, missing rate calculation, high-missing feature removal, low-IV filtering, high-PSI removal, null importance denoising, high-correlation removal, and Excel report export in sequence.
• Organization-level analysis: Computes per-organization sample counts and bad sample rates, supports out-of-sample (OOS) organization separation, and applies per-organization thresholds for IV distribution, PSI stability, and value ratio distribution.
• Statistical variable screening: Removes features with missing rates above threshold (default 0.6), overall IV below 0.1, unstable PSI exceeding threshold across months/organizations, and high correlations based on original gain — each step configurable via parameters.
• Null importance denoising: Uses label permutation with configurable tree count (n_estimators=100), depth (max_depth=5), and gain threshold (gain_threshold=50) to identify and remove noise features that don't contribute meaningful signal.
• Non-destructive with full reporting: Executes each step independently without deleting original data, and exports a comprehensive Excel cleaning report containing details and statistics from all 11 processing steps.

Who is this for?

This skill is designed for data scientists and risk analysts working in credit risk modeling who need to preprocess raw lending data before building predictive models. It is especially valuable for teams handling multi-organization credit portfolios, where automated variable screening across organizations, time periods, and statistical thresholds is essential for building robust, stable credit scoring models.