data-analysis

Data Analysis Workflow

Follow this structured approach when exploring or analysing data.

1. Define the question

Before touching data, write down in plain language:

• What question is being answered?
• Who will act on the answer, and how?
• What would "good enough" accuracy look like?

2. Understand the data

import pandas as pd
import numpy as np

df = pd.read_parquet("data/raw/...")  # or csv, SQL, etc.

# Shape and types
print(df.shape)
print(df.dtypes)

# Missing values
print(df.isnull().sum().sort_values(ascending=False))
print((df.isnull().mean() * 100).round(2))

# Basic stats
print(df.describe(include="all"))

Check:

• Row and column counts match expectations
• Dtypes are correct (dates parsed as dates, not strings)
• Missing rate per column — >30% missing warrants investigation

3. Univariate distributions

For each feature of interest:

• Numeric: histogram, box plot, check for outliers (IQR rule), skewness
• Categorical: value counts, check for near-duplicate categories (typos, casing)
• Temporal: plot over time to spot seasonality, gaps, or sudden shifts

import matplotlib.pyplot as plt
import seaborn as sns

# Numeric distribution
sns.histplot(df["column"], kde=True)
plt.title("Distribution of <column>")
plt.xlabel("<unit>")

# Categorical counts
df["category"].value_counts().head(20).plot(kind="bar")

4. Bivariate / correlation analysis

# Correlation matrix (numeric features)
corr = df.select_dtypes("number").corr()
sns.heatmap(corr, annot=True, fmt=".2f", cmap="RdYlGn", center=0)

# Feature vs target (regression)
sns.scatterplot(x="feature", y="target", data=df, alpha=0.3)

# Feature vs target (classification)
sns.boxplot(x="target_class", y="feature", data=df)

Flag any feature with correlation > 0.95 with another feature (multicollinearity risk).

5. Temporal analysis (if applicable)

df = df.sort_values("timestamp")

# Resample to daily/weekly
ts = df.set_index("timestamp")["metric"].resample("D").sum()
ts.plot(title="<Metric> over time")
plt.axvline(x=<incident_date>, color="red", linestyle="--", label="Event")

6. Document findings

End every analysis notebook with a Findings cell:

## Findings

**Question**: <restate it>

**Key observations**:
1. X% of records have missing Y — likely due to Z
2. Feature A and Feature B are strongly correlated (r=0.92)
3. There is a clear seasonal pattern in Q4

**Recommended next step**: <one concrete action>

**Caveats**: <what this analysis can't answer>

Quality checklist

• Seeds set before any random operation
• No data leakage — test set never inspected during EDA
• All charts have titles, axis labels, and units
• Missing value handling is explicit and logged
• Notebook runs clean from top to bottom (Restart & Run All)