Data Collection and Data Pre-Processing

What is the primary objective of Data Pre-Processing

Raw real-world data is typically incomplete (missing values), noisy (outliers, errors), and inconsistent. Data pre-processing cleans, transforms, and reduces data into a coherent format suitable for machine learning models.

Name three common mechanisms of Missing Data

1. MCAR (Missing Completely at Random): Missingness is entirely independent of observed and unobserved data.
2. MAR (Missing at Random): Missingness systematically depends on observed variables but not unobserved ones.
3. MNAR (Missing Not at Random): Missingness depends directly on the unobserved value itself.

What is Mean/Median Imputation and when is Median preferred

Replacing missing numerical values with the mean or median of observed values. Median is preferred when the feature distribution is skewed or contains extreme outliers, as mean is sensitive to extreme values.

Min-Max Feature Scaling and its mathematical formula

Linear rescaling of data into a fixed range, usually [0, 1]: $x_{scaled} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$

Z-score Standardization and its mathematical formula

Rescaling features so that they have a mean µ = 0 and standard deviation \sigma = 1: $z = \frac{x - µ}{\sigma}$

What is One-Hot Encoding and what issue does it introduce in high-cardinality features

Converting categorical levels into binary vector columns. For high-cardinality features, it causes curse of dimensionality and extreme sparse matrix overhead ('dummy variable trap').

What is Data Discretization

The transformation of continuous numerical attributes into discrete interval or categorical bins (e.g., converting Age into Child, Youth, Adult, Senior).

Equal-Width and Equal-Frequency Binning

• Equal-Width Binning: Divides the range [x_{\min}, x_{\max}] into k intervals of identical width w = (x_{\max} - x_{\min})/k.
• Equal-Frequency Binning: Divides sorted data into k intervals such that each bin contains roughly N/k sample records.

What is Principal Component Analysis (PCA) used for in data reduction

PCA is an unsupervised linear dimensionality reduction technique that orthogonalizes correlated features into principal components ranked by explained variance, maximizing information retention while reducing dimensions.

What is Data Deduplication in ETL pipelines

The identification and merging or deletion of duplicate, redundant, or near-identical records using primary keys, fuzzy string matching (Levenshtein distance), or record linkage algorithms.

Outlier Detection Techniques: IQR Rule vs Z-Score Method

Outlier Detection Foundations

1. IQR (Interquartile Range) Method (Tukey's Fences):

• IQR = Q_3 - Q_1
• Lower Bound = Q_1 - 1.5 × IQR
• Upper Bound = Q_3 + 1.5 × IQR
• Non-parametric; highly robust against extreme skewness.

1. Z-Score Method (Parametric):

• z = \frac{x - µ}{\sigma}
• A point is classified as an outlier if |z| > 3 (outside 99.7% confidence interval under Gaussian distribution).
• Sensitive to sample size and heavily skewed distributions.

Compare Strategies for Handling Missing Values: Deletion vs Imputation

Detail the Process and Need for Data Discretization and Concept Hierarchies

Data Discretization Architecture

Continuous data often contains noise and micro-variations that degrade rule induction and tree-based performance. Discretization constructs compact representations:

1. Unsupervised Discretization:

• Equal-width binning (sensitive to outliers).
• Equal-frequency binning (good for non-uniform data).
• K-means clustering-based binning.

1. Supervised Discretization:

• Entropy-based discretization (MDLP): Maximizes information gain between target classes and bin boundaries.
• ChiMerge: Uses \chi^2 statistics to iteratively merge adjacent intervals that exhibit identical class distributions.

1. Concept Hierarchies: Rolling up atomic data into higher abstractions (Street \to City \to State \to Country).

Data Integration Challenges: Schema Integration, Redundancy, and Entity Identification

Integrating heterogeneous enterprise databases presents three core technical hurdles:

1. Entity Identification Problem: Matching real-world entities across disparate sources where schema attributes differ (e.g., cust_id vs client_no, or fuzzy name matching A. Sharma vs Akshay Sharma).
2. Schema Conflicts: Structural divergence (e.g., currency in USD vs INR, imperial vs metric units, date formats YYYY-MM-DD vs DD/MM/YYYY).
3. Redundancy & Correlation Analysis: Redundant attributes inflate computational load. Detected via Pearson's Product-Moment Coefficient for numerical variables or Chi-Square ($\chi^2$) Contingency Tests for nominal attributes.

Feature Selection and Feature Extraction in Data Reduction

Data Reduction Taxonomy

• Feature Selection: Chooses an optimal subset d from original D features without altering their original semantics. Methods include:
• Filter Methods: Pearson correlation, ANOVA F-test, Mutual Information.
• Wrapper Methods: Recursive Feature Elimination (RFE), Forward Stepwise Selection.
• Embedded Methods: LASSO (L_1 regularization) driving non-essential coefficients to exact zero.
• Feature Extraction: Transforms the original high-dimensional space into a new, lower-dimensional synthetic coordinate space (e.g., PCA, LDA, t-SNE, Autoencoders). Preserves variance while losing direct human interpretability of raw features.

Provide a Complete Mathematical and Algorithmic Derivation of Principal Component Analysis (PCA) for Dimensionality Reduction

Mathematical Derivation of PCA

1. Standardization

Given dataset X \in \mathbb{R}^{N × D}, calculate column-wise means µ_j and zero-center: \tilde{x}_{ij} = x_{ij} - µ_j.

2. Covariance Matrix Calculation

$\Sigma = \frac{1}{N} \tilde{X}^T \tilde{X} \quad (D × D symmetric positive semi-definite matrix) where \Sigma{jk} = \text{Cov}(Xj, X_k)$.

3. Eigendecomposition

Solve the characteristic equation for eigenvalues \lambda and eigenvectors v: $\det(\Sigma - \lambda I) = 0 \implies \Sigma v = \lambda v$

4. Selection and Explained Variance

Sort eigenvalues in descending order: \lambda_1 ≥ \lambda_2 ≥ \dots ≥ \lambda_D. The fraction of total variance explained by the top k components is: $Explained Variance Ratio = \frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^D \lambda_j}$

5. Orthogonal Projection

Form transformation matrix W_k = [v_1, v_2, \dots, v_k] \in \mathbb{R}^{D × k}. The dimensionally reduced dataset is: $Z = \tilde{X} W_k \in \mathbb{R}^{N × k}$

Design and Implement an Industrial-Strength Data Cleaning Pipeline in Python using Pandas and Scikit-Learn

Production Data Pre-Processing Implementation

Architecture Principles:

1. Zero Data Leakage: Imputation statistics (median) and scaling quantiles are fit strictly on training splits.
2. Robustness to Extreme Outliers: RobustScaler scales data according to the IQR, preventing extreme values from compressing standard deviation.

Critically Analyze Missing Data Mechanisms (MCAR, MAR, MNAR) and Modern Multiple Imputation by Chained Equations (MICE)

Theoretical Framework of Missing Data & MICE

1. Rubin's Typology of Missing Data

• MCAR: P(M | Y_{obs}, Y_{mis}) = P(M). Example: Laboratory blood sample tube drops and shatters accidentally.
• MAR: P(M | Y_{obs}, Y_{mis}) = P(M | Y_{obs}). Example: Elderly patients are less likely to report income, but conditional on age, missingness is random.
• MNAR: P(M | Y_{obs}, Y_{mis}) depends directly on Y_{mis}. Example: High-income individuals refusing to disclose net wealth due to taxation privacy concerns.

2. The MICE Algorithm (Chained Equations)

MICE operates under the Fully Conditional Specification (FCS) paradigm:

1. Step 1: Initialize missing values with random draws from observed data.
2. Step 2: For each variable Y_j with missing values (j = 1, \dots, p):

• Set Y_j back to missing.
• Fit a regression model of Y_j on all other variables Y_{-j} using observed cases: Y_j \sim f(Y_1, \dots, Y_{j-1}, Y_{j+1}, \dots, Y_p).
• Impute missing values in Y_j by sampling from the posterior predictive distribution of the fitted model.

1. Step 3: Repeat Step 2 cyclically for T iterations (typically 10–20) until convergence is achieved.
2. Step 4: Generate m distinct imputed datasets to account for imputation uncertainty, run downstream modeling on all m datasets, and pool parameters using Rubin's Rules.

Examiners frequently test: IQR outlier detection rule (5 marks), Min-Max vs Z-Score formulas (2/5 marks), and PCA mathematical steps (10/15 marks).