Unit 3: Exploratory Data Analytics (EDA) — Introduction to Data Science

Core Definitions & Key Concepts

Descriptive Statistics versus Inferential Statistics

Descriptive Statistics quantitatively summarizes, describes, and visualizes properties of an observed sample. Inferential Statistics uses sample data to deduce properties and draw probabilistic conclusions about an unobserved broader population.

State the mathematical definitions of Mean, Median, and Mode

Mean: \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i
Median: The middle value separating the upper half from the lower half of sorted data.
Mode: The most frequently occurring value in the dataset.

Sample Variance and Standard Deviation with formulas

Sample Variance: s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2
Standard Deviation: s = \sqrt{s^2} (The degree of dispersion expressed in original units).

What does Skewness measure and what does positive skewness indicate

Skewness measures the asymmetry of a distribution around its mean. Positive (right) skewness indicates a longer tail extending towards higher values, where Mean > Median > Mode.

What is Kurtosis and what is a Leptokurtic distribution

Kurtosis measures the 'tailedness' and outlier-proneness of a probability distribution. A Leptokurtic distribution (Kurtosis > 3) possesses fatter tails and a sharper central peak than a normal Gaussian distribution.

The 5-number summary depicted in a standard Box Plot

Minimum (Q_1 - 1.5 × IQR)
First Quartile (Q_1 or 25th percentile)
Median (Q_2 or 50th percentile)
Third Quartile (Q_3 or 75th percentile)
Maximum (Q_3 + 1.5 × IQR)

Pearson's Correlation Coefficient (r) and state its valid range

Measures linear relationship strength and direction between two continuous variables: $r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}, \quad -1 ≤ r ≤ +1$

What is the primary difference between Pearson and Spearman rank correlation

Pearson measures purely linear relationships between continuous normal variables. Spearman assesses monotonic relationships (whether linear or non-linear) by calculating Pearson correlation on rank orders, resisting outliers.

What does a Heat Map represent in exploratory data analysis

A two-dimensional graphical representation of data where values in a matrix (e.g., cross-feature correlation matrix) are encoded as distinct color gradients, allowing rapid visual pattern detection.

What is the primary hypothesis tested by One-Way ANOVA (Analysis of Variance)

Tests the null hypothesis (H_0) that three or more population means are identical: H_0: µ_1 = µ_2 = \dots = µ_k against the alternative hypothesis (H_1) that at least one group mean is significantly different.

✓Topic 1 covered: Core Definitions & Key Concepts

Next: Conceptual & Architectural Analysis↓

Conceptual & Architectural Analysis

Compare Measures of Central Tendency and Dispersion Under Skewed Data

Metric	Definition	Robust to Outliers?	Behavior in Right-Skewed Data
Mean	Center of gravity of values	❌ No (Heavily pulled)	Substantially inflated (\bar{x} > Median)
Median	50th percentile value	✅ Yes (Resistant)	Represents typical central observation
Mode	Highest frequency peak	✅ Yes (Resistant)	Remains at highest frequency cluster
Std. Dev.	Root mean squared dev.	❌ No (Inflated by square)	Overestimates dispersion of typical data
IQR	Spread of central 50% (Q_3 - Q_1)	✅ Yes (Resistant)	Accurately describes interquartile spread

Skewness and Kurtosis with Formulas, Diagrams, and Interpretation

Mathematical Formulations

Sample Skewness (Fisher-Pearson coefficient):

$g_1 = \frac{n}{(n-1)(n-2)} \sum_{i=1}^n ≤ft( \frac{x_i - \bar{x}}{s} \right)^3$

g_1 = 0: Symmetrical (Gaussian normal).
g_1 > 0: Positive (Right-skewed, tail to the right).
g_1 < 0: Negative (Left-skewed, tail to the left).

Excess Kurtosis:

$g_2 = \frac{n(n+1)}{(n-1)(n-2)(n-3)} \sum_{i=1}^n ≤ft( \frac{x_i - \bar{x}}{s} \right)^4 - \frac{3(n-1)^2}{(n-2)(n-3)}$

Mesokurtic ($g_2 = 0$): Normal distribution baseline.
Leptokurtic ($g_2 > 0$): Heavy tails, high outlier density (e.g., financial market returns).
Platykurtic ($g_2 < 0$): Light tails, flat peak (e.g., uniform distribution).

The Architecture and Utility of Pivot Tables and Cross-Tabulations in Business EDA

Pivot Table Architecture

A Pivot Table dynamically summarizes multi-dimensional categorical and numerical data through grouping, pivoting, and aggregation.

System Architecture / Concept DiagramHorizontal Scroll Enabled

+------------------+-----------------------------+
|                  |       Region (Column)       |
| Product (Row)    | North    South   East   West|
+------------------+-----------------------------+
| Laptops          | $420k    $310k   $280k  $510k|
| Smartphones      | $890k    $740k   $620k  $930k|
+------------------+-----------------------------+
| Total Revenue    | SUM(sales), AVG(margin)     |
+------------------+-----------------------------+

Core Operations: Filtering, column-to-row pivoting, multi-level hierarchy aggregation (pd.pivot_table(df, values='sales', index=['category'], columns=['region'], aggfunc='mean')).

Correlation Analysis: Pitfalls, Collinearity, and 'Correlation does not imply Causation'

Correlation Pitfalls in Data Science

Spurious Correlation: Two independent variables exhibiting high correlation due to an unobserved confounding variable (e.g., Ice cream sales and drowning rates both correlate positively with high summer temperatures).
Anscombe's Quartet: Four distinct datasets possessing identical means (9.0), variances (11.0), and Pearson correlations (r = 0.816), yet exhibiting radically different geometric structures when visualized.
Multicollinearity: When two or more predictor features in regression are highly linearly correlated (|r| > 0.85), inflating the variance of coefficient estimates (VIF > 5) and destabilizing model interpretations.

Detail the Mathematical Theory and Application of One-Way ANOVA

One-Way ANOVA Hypothesis Framework

ANOVA partitions total variation in data into two orthogonal components: between-group variance and within-group variance.

Total Sum of Squares ($SST$):

$SST = \sum_{j=1}^k \sum_{i=1}^{n_j} (x_{ij} - \bar{x}_{..})^2, \quad df = N - 1$

Between-Group Sum of Squares ($SSB$):

$SSB = \sum_{j=1}^k n_j (\bar{x}_{.j} - \bar{x}_{..})^2, \quad df = k - 1$

Within-Group Sum of Squares ($SSW$):

$SSW = \sum_{j=1}^k \sum_{i=1}^{n_j} (x_{ij} - \bar{x}_{.j})^2, \quad df = N - k$

F-Statistic Calculation:

$F = \frac{MSB}{MSW} = \frac{SSB / (k - 1)}{SSW / (N - k)} If F{\text{calc}} > F{\text{critical}}(\alpha, k-1, N-k), reject H_0$ and conclude that at least one group mean significantly differs.

✓Topic 2 covered: Conceptual & Architectural Analysis

Next: In-Depth Analysis & Implementation↓

In-Depth Analysis & Implementation

Design and Implement an Exhaustive Exploratory Data Analysis (EDA) Script in Python Using Seaborn and Matplotlib for an E-Commerce Dataset

Comprehensive EDA Python Implementation

Python

1import matplotlib.pyplot as plt
2import numpy as np
3import pandas as pd
4import scipy.stats as stats
5import seaborn as sns
6
7def perform_comprehensive_eda(df, numerical_cols, categorical_cols):
8    """
9    Executes a structured university-grade EDA workflow:
10    1. Descriptive Statistical Matrix
11    2. Univariate Distribution & Skewness Profiling
12    3. Bivariate Correlation Matrix & Heatmap
13    4. Categorical ANOVA Hypothesis Testing
14    """
15    print("=== 1. DESCRIPTIVE STATISTICAL PROFILE ===")
16    stats_df = df[numerical_cols].describe().T
17    stats_df['skewness'] = df[numerical_cols].skew()
18    stats_df['kurtosis'] = df[numerical_cols].kurtosis()
19    print(stats_df[['mean', 'std', '50%', 'skewness', 'kurtosis']])
20    
21    # 2. Correlation Matrix Heatmap
22    plt.figure(figsize=(10, 8))
23    corr_matrix = df[numerical_cols].corr(method='pearson')
24    mask = np.triu(np.ones_like(corr_matrix, dtype=bool))
25    sns.heatmap(corr_matrix, mask=mask, annot=True, fmt=".2f", cmap='coolwarm',
26                vmin=-1, vmax=1, square=True, linewidths=0.5)
27    plt.title('Bivariate Correlation Heatmap (Lower Triangle)', fontsize=14)
28    plt.tight_layout()
29    plt.savefig('correlation_heatmap.png', dpi=300)
30    plt.close()
31    
32    # 3. Box Plot Anomaly Detection
33    plt.figure(figsize=(12, 6))
34    for idx, col in enumerate(numerical_cols[:4], 1):
35        plt.subplot(1, 4, idx)
36        sns.boxplot(y=df[col], color='skyblue')
37        plt.title(f'Box Plot: {col}')
38    plt.tight_layout()
39    plt.savefig('boxplots_distribution.png', dpi=300)
40    plt.close()
41    print("EDA Visualizations generated successfully.")

Workflow Highlights:

Triangular Correlation Mask: Prevents redundant visual clutter by masking the duplicate upper half of the symmetric matrix.
Skewness Integration: Concurrently profiles statistical moments alongside standard quantiles.

The Complete Statistical Machinery of ANOVA: Construction of ANOVA Table, F-Distribution, and Post-Hoc Tukey HSD Tests

Complete ANOVA Statistical Mechanics

System Architecture / Concept DiagramHorizontal Scroll Enabled

+---------------------------------------------------------------------------------------------------------+
|                                         STANDARD ANOVA TABLE                                            |
+---------------------+-------------------+---------------------+--------------------+--------------------+
| Source of Variation | Sum of Squares SS | Degrees of Freedom  | Mean Square (MS)   | F-Statistic        |
+---------------------+-------------------+---------------------+--------------------+--------------------+
| Between Groups      | SSB               | k - 1               | MSB = SSB / (k-1)  | F = MSB / MSW      |
| Within Groups       | SSW               | N - k               | MSW = SSW / (N-k)  |                    |
| Total               | SST               | N - 1               |                    |                    |
+---------------------+-------------------+---------------------+--------------------+--------------------+

1. Fundamental Assumptions of ANOVA

Normality: Residuals within each treatment group are normally distributed: \epsilon_{ij} \sim \mathcal{N}(0, \sigma^2). Evaluated via Shapiro-Wilk or Q-Q plots.
Homogeneity of Variance (Homoscedasticity): Variance across all groups is identical: \sigma_1^2 = \sigma_2^2 = \dots = \sigma_k^2. Evaluated via Levene's Test or Bartlett's Test.
Independence: Observations are sampled independently.

2. Post-Hoc Analysis: Tukey's Honestly Significant Difference (HSD)

When ANOVA rejects the null hypothesis (p < 0.05), it fails to pinpoint which specific pairs of group means differ. Tukey's HSD calculates pairwise Studentized range statistics: $q = \frac{\bar{x}_i - \bar{x}_j}{\sqrt{\frac{MSW}{2} ≤ft( \frac{1}{n_i} + \frac{1}{n_j} \right)}} If |q| > q_{\text{crit}}(\alpha, k, N-k), the difference between group i and group j$ is statistically significant, strictly controlling family-wise error rate (FWER).

Analyze Modern Visual Exploratory Paradigms: Violin Plots, Pair Plots, Q-Q Plots, and Dimensionality Reduction Visualizations

Advanced Visual Analytics Taxonomy

1. Violin Plot vs. Box Plot

A Box Plot depicts only five summary numbers and outliers, completely masking multi-modal distributions (e.g., bimodal clusters look identical to uniform spreads).
A Violin Plot combines a miniature box plot with a rotated Kernel Density Estimation (KDE) on each side, revealing probability densities, multi-modal clusters, and distribution shapes simultaneously.

2. Normal Quantile-Quantile (Q-Q) Plot

Plots the quantiles of sample data against the theoretical quantiles of a standard Gaussian normal distribution \mathcal{N}(0, 1).
Interpretation: If points fall along the straight diagonal 45^\circ line, normality is established. Upward curvature at tails indicates heavy right tails (leptokurtic); S-shaped curvature indicates bounded platykurtic spreads.

3. t-Distributed Stochastic Neighbor Embedding (t-SNE)

Non-linear probabilistic technique converting Euclidean distances between data points into conditional probabilities representing similarities, minimizing Kullback-Leibler (KL) divergence in low-dimensional space for visual cluster identification.

✓Topic 3 covered: In-Depth Analysis & Implementation

Next: Important Exam Points & Quick Revision↓

Important Exam Points & Quick Revision

Examiners frequently test: Box plot 5-number summary (5 marks), Pearson correlation formula & range (2/5 marks), and One-way ANOVA Table construction (10/15 marks).

Practice Ready

Ready to practice Unit 3?

You've read the core concepts. Take the balanced 10-question practice set to test your mastery!

Save progress