Exploratory
Data Analysis

Jon Reades - j.reades@ucl.ac.uk

1st October 2026

Epicyclic Feedback

Peng and Matsui, The Art of Data Science, p.8

	Set Expectations	Collect Information	Revise Expectations
Question	Question is of interest to audience	Literature search/experts	Sharpen question
EDA	Data are appropriate for question	Make exploratory plots	Refine question or collect more data
Modelling	Primary model answers question	Fit secondary models / analysis	Revise model to include more predictors
Interpretation	Interpretation provides specific and meaningful answer	Interpret analyses with focus on effect and uncertainty	Revise EDA and/or models to provide more specific answers
Communication	Process & results are complete and meaningful	Seek feedback	Revises anlyses or approach to presentation

Approaching EDA

There’s no hard and fast way of doing EDA, but as a general rule you’re looking to:

• Clean
• Canonicalise
• Clean More
• Visualise & Describe
• Review
• Clean Some More
• …

The ‘joke’ is that 80% of Data Science is data cleaning.

Cleaning Part 1: testing validity of records (possibly while tracking rejected records for subsequent analysis)

Canonicalisation: controling for variation (e.g. typos, capitalisation, formatting, leading/trailing whitespace, different types of NULL values, etc.) and in a spatial context deal with projection and geo-data issues.

Cleaning Part 2: further testing of records (e.g. deciding what to do with NaNs, missing values, outside of study area, etc.)

Visualise & Describe: covered in QM but we’ll take a high-level look at this.

A Related Take

EDA—Don’t ask how, ask what:

• Descriptive Statistics: get a high-level understanding of your dataset.
• Missing values: come to terms with how bad your dataset is.
• Distributions and Outliers: and why countries that insist on using different units make our jobs so much harder.
• Correlations: and why sometimes even the most obvious patterns still require some investigating.

Another Take

Here’s another view of how to do EDA:

1. Preview data randomly and substantially
2. Check totals such as number of entries and column types
3. Check nulls such as at row and column levels
4. Check duplicates: do IDs recurr, did the servers fail
5. Plot distribution of numeric data (univariate and pairwise joint distribution)
6. Plot count distribution of categorical data
7. Analyse time series of numeric data by daily, monthly and yearly frequencies

In Practice

import pandas as pd
import geopandas as gpd
url='https://bit.ly/3I0XDrq'
df = pd.read_csv(url)
df.set_index('id', inplace=True)
df['price'] = df.price.str.replace('$','',regex=False).astype('float')
gdf = gpd.GeoDataFrame(df,
            geometry=gpd.points_from_xy(
                        df['longitude'],
                        df['latitude'],
                        crs='epsg:4326'
            )
      )

Getting Started

You can follow along by loading the Inside Airbnb sample:

import pandas as pd
import geopandas as gpd
url='https://bit.ly/3I0XDrq'
df = pd.read_csv(url)
df.set_index('id', inplace=True)
df['price'] = df.price.str.replace('$','',regex=False).astype('float')
gdf = gpd.GeoDataFrame(df, 
            geometry=gpd.points_from_xy(
                        df['longitude'], 
                        df['latitude'], 
                        crs='epsg:4326'
            )
      )

Note that this (re)loads the sampled Airbnb data from GitHub every time you run it. For a large data set on someone else’e server you might want to save and (re)load it locally. A simple helper function would then just check if the file already existed locally before trying to download the file again: that would allow you to work while offline and speed up your code substantially too!

What Can We Do?¹ (Series)

print(f"Host count is {gdf.host_name.count()}")
print(f"Mean is {gdf.price.mean():.0f}")
print(f"Max price is {gdf.price.max()}")
print(f"Min price is {gdf.price.min()}")
print(f"Median price is {gdf.price.median()}")
print(f"Standard dev is {gdf.price.std():.2f}")
print(f"25th quantile is {gdf.price.quantile(q=0.25)}")

Host count is 100
Mean is 118
Max price is 798.0
Min price is 15.0
Median price is 80.5
Standard dev is 123.87
25th quantile is 40.75

1. This is by no means all that we can do…

What Can We Do?¹ (Data Frame Edition)

Command	Returns
print(df.mean()) print(df.count()) print(df.max()) # ... print(df.corr()) print(df.describe())	Mean of each *numerical* column Number of non-null values in each column Highest value in each column ... Correlation between columns Summarise

Notice how we have the same functionality, but it operates at the level of the data set itself now. We gain a few new functions as well that relate to interactions between columns (a.k.a. data series).

1. Hint: there’s a lot more we could do!

Measures

So pandas provides functions for commonly-used measures:

print(f"Mean: {df.price.mean():.2f}")
print(f"Median: {df.price.median():.2f}")
print(f"25th Percentile: {df.price.quantile(0.25):.2f}")

Mean: 118.45
Median: 80.50
25th Percentile: 40.75

More Complex Measures

Pandas also makes it easy to derive new variables… Here’s the z-score:

df['zscore'] = (df.price - df.price.mean())/df.price.std()
df.plot.box(column='zscore', figsize=(8,4.5))

And Even More Complex

And here’s the Inter-Quartile Range Standardised score:

df['iqr_std'] = (df.price - df.price.median())/ \
      (df.price.quantile(q=0.75)-df.price.quantile(q=0.25))
df.plot.box(column='iqr_std', figsize=(8,4.5))

The Plot Thickens

We’ll get to more complex plotting over the course of the term, but here’s a good start for exploring the data! All plotting depends on matplotlib which is the ogre in the attic to R’s ggplot.

import matplotlib.pyplot as plt

Get used to this import as it will allow you to save and manipulate the figures created in Python. It is not the most intuitive approach (unless you’ve used MATLAB before) but it does work.

Confession Time

I do like ggplot and sometimes even finish off graphics for articles in R just so that I can use ggplot; however, it is possible to generate great-looking figures in matplotlib but it is often more work because it’s a lot less intuitive.

Boxplot

df.price.plot.box(figsize=(8,4.5))
plt.savefig('pboxplot.png', dpi=150, transparent=True)

Frequency

df.room_type.value_counts().plot.bar(figsize=(8,4.5))
plt.savefig('phistplot.png', dpi=150, transparent=True)

A Correlation Heatmap

We’ll get to these in more detail in a couple of weeks, but here’s some output…

import numpy as np
cols = df.select_dtypes(
    include=np.number
).columns.to_list()[:4]
plt.matshow(df[cols].corr())
xaxis = np.arange(len(cols))
ax = plt.gca()
ax.set_xticks(xaxis)
ax.set_yticks(xaxis)
ax.set_xticklabels(cols)
ax.set_yticklabels(cols)

A ‘Map’

df.plot.scatter(x='longitude', y='latitude', figsize=(8,4.5))
plt.savefig('pscatterplot.png', dpi=150, transparent=True)

A Fancy ‘Map’

df.plot.scatter(x='longitude',y='latitude',
                c='price',colormap='viridis',
                figsize=(8,4.5),title='London',
                grid=True,s=24,marker='x')
plt.savefig('pscatterplot.png', dpi=150, transparent=True)

An Actual ‘Map’

gdf.plot(column='price', cmap='viridis', 
         scheme='quantiles', markersize=8, legend=True,
         figsize=(8,4.5))

Additional Resources

There’s so much more to find, but:

• Pandas Reference
• A Guide to EDA in Python (Looks very promising)
• EDA with Pandas on Kaggle
• EDA Visualisation using Pandas
• Python EDA Analysis Tutorial
• Better EDA with Pandas Profiling [Requires module installation]
• EDA: DataPrep.eda vs Pandas-Profiling [Requires module installation]

• A Data Science Project for Beginners (EDA)
• EDA: A Pracitcal Guide and Template for Structured Data
• EDA—Don’t ask how, ask what (Part 1)
• Preparing your Dataset for Modeling – Quickly and Easily (Part 2)
• Handling Missing Data
• Introduction to Exploratory Data Analysis (EDA)

Thank You

References