Model Development

What is the core premise of Simple Linear Regression

Simple Linear Regression models the relationship between a single independent continuous variable x and a dependent continuous variable y using a linear equation: $y = \beta_0 + \beta_1 x + \epsilon$

Ordinary Least Squares (OLS) criterion

OLS minimizes the sum of squared vertical differences (residuals) between observed points y_i and predicted values \hat{y}_i: $SSE = \sum_{i=1}^n (y_i - \hat{y}_i)^2$

State the four classical Gauss-Markov assumptions of linear regression

1. Linearity in parameters
2. Strict exogeneity: E[\epsilon | X] = 0
3. Homoscedasticity and no autocorrelation: Var(\epsilon_i | X) = \sigma^2, Cov(\epsilon_i, \epsilon_j) = 0
4. No perfect multicollinearity in X

Multiple Linear Regression in matrix notation

$Y = X\beta + \epsilon where Y \in \mathbb{R}^{n \times 1}, X \in \mathbb{R}^{n \times (p+1)}, \beta \in \mathbb{R}^{(p+1) \times 1}, and \epsilon \in \mathbb{R}^{n \times 1}$.

What is the closed-form Normal Equation for OLS parameter estimation

$\hat{\beta} = (X^T X)^{-1} X^T Y$

A Residual in regression analysis

The difference between the actual observed value y_i and the predicted value \hat{y}_i: $e_i = y_i - \hat{y}_i$

What does a Residual Plot show and what indicates a well-fitted model

A scatter plot of residuals e_i on the vertical axis against predicted values \hat{y}_i or independent variable x on the horizontal axis. A random, uniform cloud of points around 0 indicates constant variance and correct linear model specification.

What is Polynomial Regression and how does it retain linear estimation properties

Models non-linear relationships by adding powers of original predictors (x^2, x^3). It is still considered a linear model because it remains linear in the unknown parameters \beta_i, allowing standard OLS estimation.

Coefficient of Determination (R^2) and state its interpretation

$R^2 = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$ Represents the proportion of total variance in the dependent variable explained by the regression model.

Why is Adjusted R^2 preferred over ordinary R^2 in Multiple Regression

Ordinary R^2 artificially increases whenever new predictors are added, even if irrelevant. Adjusted $R^2$ penalizes model complexity by factoring in degrees of freedom: R^2_{adj} = 1 - ≤ft[ \frac{(1-R^2)(n-1)}{n - p - 1} \right].

Derive the Closed-Form Analytical Solution for Simple Linear Regression Slope and Intercept

OLS Simple Linear Derivation

Minimize the sum of squared errors: $S(\beta_0, \beta_1) = \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i)^2$

1. Partial Derivative with respect to $\beta_0$:

$\frac{\partial S}{\partial \beta_0} = -2 \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i) = 0 \implies n\beta_0 = \sum y_i - \beta_1 \sum x_i \implies \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$

1. Partial Derivative with respect to $\beta_1$:

$\frac{\partial S}{\partial \beta_1} = -2 \sum_{i=1}^n x_i (y_i - \beta_0 - \beta_1 x_i) = 0 Substitute \hat{\beta}0 = \bar{y} - \hat{\beta}1 \bar{x}: \sum x_i (y_i - \bar{y} + \hat{\beta}_1 \bar{x} - \hat{\beta}_1 x_i) = 0 \implies \sum (x_i - \bar{x})(y_i - \bar{y}) = \hat{\beta}_1 \sum (x_i - \bar{x})^2 \hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{Cov(X, Y)}{Var(X)}$

In-Sample Evaluation Metrics: MAE, MSE, RMSE, and MAPE

Residual Diagnostics: Heteroscedasticity, Non-Linearity, and Autocorrelation

Residual Diagnostic Patterns

• Heteroscedasticity: Unequal variance of residuals across levels of predicted \hat{y}. Breaches OLS efficiency. Detected via Breusch-Pagan or White Test.
• Non-Linearity: Curved U-shape in residual plot indicates missing polynomial or interaction terms.
• Autocorrelation: Residuals correlate with lagged residuals (common in time series). Detected via Durbin-Watson Test (ideal value d ≈ 2.0).

Detail the Concept and Implementation of Scikit-Learn Machine Learning Pipelines

Machine Learning Pipeline Principles

In enterprise systems, data transformations (imputation, polynomial expansion, scaling) must be strictly coupled with model estimation into an atomic Pipeline:

1. Encapsulation: Guarantees that validation/test folds are transformed strictly using parameters learned from training folds, eliminating data leakage.
2. Code Maintainability: Replaces fragmented preprocessing scripts with a single unified .fit() and .predict() interface.
3. Hyperparameter Tuning: Enables automated grid searching over preprocessing options and estimator hyperparameters simultaneously.

Simple, Multiple, and Polynomial Regression

Regression Model Paradigm Matrix

• Simple Linear Regression: Fits a line y = \beta_0 + \beta_1 x in 2D space. Limited to single-input linear problems.
• Multiple Linear Regression: Fits a hyper-plane y = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p in (p+1)-dimensional space. Captures multi-factor influences and controls for confounding covariates.
• Polynomial Regression: Fits curved surfaces y = \beta_0 + \beta_1 x + \beta_2 x^2 + \dots + \beta_d x^d. Accurately models non-linear curvilinear phenomena (e.g., diminishing returns), but risks severe overfitting when degree d is excessive.

Provide the Complete Mathematical Derivation of the Normal Equations for Multiple Linear Regression Using Matrix Calculus

Matrix Derivation of OLS Estimator

Let the Multiple Linear Regression model be represented as: $Y = X\beta + \epsilon where: Y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}, \quad X = \begin{bmatrix} 1 & x_{11} & \dots & x_{1p} \\ 1 & x_{21} & \dots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & \dots & x_{np} \end{bmatrix}, \quad \beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix}, \quad \epsilon = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix}$

1. Formulate Residual Sum of Squares (SSE)

The residual vector is e = Y - X\hat{\beta}. $SSE(\hat{\beta}) = e^T e = (Y - X\hat{\beta})^T (Y - X\hat{\beta}) Expand the quadratic matrix product: SSE(\hat{\beta}) = (Y^T - \hat{\beta}^T X^T)(Y - X\hat{\beta}) = Y^T Y - Y^T X \hat{\beta} - \hat{\beta}^T X^T Y + \hat{\beta}^T X^T X \hat{\beta} Since Y^T X \hat{\beta} is a scalar (1 \times 1), its transpose equals itself: (Y^T X \hat{\beta})^T = \hat{\beta}^T X^T Y. SSE(\hat{\beta}) = Y^T Y - 2 \hat{\beta}^T X^T Y + \hat{\beta}^T X^T X \hat{\beta}$

2. Differentiate with Respect to Vector \hat{\beta}

Apply standard vector calculus identities: $\frac{\partial (a^T x)}{\partial x} = a, \quad \frac{\partial (x^T A x)}{\partial x} = 2Ax \quad (for symmetric A) Set the gradient to zero vector \mathbf{0}: \nabla_{\hat{\beta}} SSE = -2 X^T Y + 2 X^T X \hat{\beta} = \mathbf{0} X^T X \hat{\beta} = X^T Y \quad (The Classical Normal Equations)$

3. Closed-Form Parameter Solution

Assuming matrix X has full column rank (X^T X is non-singular and invertible): $\hat{\beta} = (X^T X)^{-1} X^T Y$

4. Second-Order Condition (Sufficiency)

The Hessian matrix is \nabla^2 SSE = 2 X^T X. Since X^T X is positive definite for linearly independent feature columns, \hat{\beta} uniquely minimizes the sum of squared errors.

Design and Build a Complete Polynomial Regression and Pipeline Workflow in Python with Residual and Distribution Diagnostics

Complete Python Implementation

Analyze the Bias-Variance Tradeoff in Regression Modeling and Mathematically Decompose Expected Generalization Error

Bias-Variance Tradeoff Analysis

1. Mathematical Decomposition of Expected Test Error

Assume true data generation process: y = f(x) + \epsilon, where E[\epsilon] = 0 and Var(\epsilon) = \sigma^2. Let \hat{f}(x) be the model estimated from training dataset \mathcal{D}. The expected squared prediction error at a test point x across all training realizations is: $MSE(x) = E≤ft[ (y - \hat{f}(x))^2 \right] Expand terms: MSE(x) = E≤ft[ (f(x) + \epsilon - \hat{f}(x))^2 \right] = E≤ft[ (f(x) - \hat{f}(x))^2 \right] + 2E[(f(x) - \hat{f}(x))\epsilon] + E[\epsilon^2] Since noise \epsilon is independent with E[\epsilon] = 0, the cross-term vanishes: = E≤ft[ (f(x) - \hat{f}(x))^2 \right] + \sigma^2 Now examine E\left[ (f(x) - \hat{f}(x))^2 \right]. Add and subtract E[\hat{f}(x)] inside the expectation: = E≤ft[ (f(x) - E[\hat{f}(x)] + E[\hat{f}(x)] - \hat{f}(x))^2 \right] = (f(x) - E[\hat{f}(x)])^2 + E≤ft[ (\hat{f}(x) - E[\hat{f}(x)])^2 \right] - 2(f(x) - E[\hat{f}(x)])E[\hat{f}(x) - E[\hat{f}(x)]] The cross-term is 0 because E[\hat{f}(x) - E[\hat{f}(x)]] = 0. \therefore Expected Prediction Error = \underbrace{(f(x) - E[\hat{f}(x)])^2}_{Bias^2} + \underbrace{E≤ft[ (\hat{f}(x) - E[\hat{f}(x)])^2 \right]}_{Variance} + \underbrace{\sigma^2}_{Irreducible Error}$

2. Architectural Implications

• High Bias (Underfitting): Simple models (e.g., straight line on cubic curve) make erroneous assumptions; unable to capture training patterns.
• High Variance (Overfitting): Complex models (e.g., 15th-degree polynomial) fit idiosyncratic training noise; generalize terribly to test datasets.

Examiners frequently test: OLS derivation and Normal Equation (10/15 marks), R2 vs Adjusted R2 (5 marks), and Residual Plot interpretation (5 marks).