Numeric Data
Overview
This week we will be introducing the use of the pandas library for data analysis and management through a focus on numeric data and its distribution(s). This marks a major shift from working with concepts (lists, dictionaries, functions, etc.) largely in isolation to encountering all of them together ‘in the wild’ as part of a full data science workflow. So we are moving from the acquisition of concepts to their integration in the same way that we will — over the course of these three sessions — be coming from data acquisition to data integration.
By the end of this week you should be able to:
- Use Boolean logic and seeded pseudo-random numbers to write code that behaves predictably and reproducibly.
- Read, clean and summarise tabular data with pandas.
- Explain how a data-driven workflow complements the statistical analysis taught in QM.
- Explain how data can hide or distort what they claim to measure (D’Ignazio and Klein (2020a, Ch.4); Cox and Slee (2016)).
Watch the four pre-recorded lectures (≈ 70 min in total): Logic (9 min) · Randomness (17 min) · Data (18 min) · Pandas (24 min)
This is a heavy week, so split it across two sittings.
Read D’Ignazio and Klein (2020a) and Cox and Slee (2016), making notes against the study guide questions.
Start organising your ideas for the briefing; once we’ve covered Pandas you can begin exploring the Inside Airbnb data.
Bring your laptop with Podman running.
Readings
Come to class prepared to discuss the following readings:
| Citation | Article |
|---|---|
| D’Ignazio and Klein (2020a) Ch.4 | URL |
| Cox and Slee (2016) | URL |
Study Guide
Reading D’Ignazio and Klein (2020a, Ch.4):
- The chapter’s principle is to “rethink binaries and hierarchies”. Using the census or Facebook examples, explain how a classification system decides what can be counted, and what this means for Anderson’s claim that, with enough data, the numbers speak for themselves.
- What is the “paradox of exposure”? Who stands to gain and who stands to lose from being counted, and why does this mean that more data (Anderson’s “more is different”) is not automatically better?
- How might the concept of the “privilege hazard” inform an analysis of Anderson’s argument?
Considering Cox and Slee (2016):
- How does the case study presented in Cox and Slee illustrate the issues explored in the other readings?
- According to Cox and Slee, how did the cleaning of the New York City listings reflect the company’s power and privilege?
- How does this cleaning potentially harm less-privileged stakeholders in the company’s platform?
Read D’Ignazio and Klein (2020b) to highlight the importance of thinking about what a data set captures… and what it excludes. Cox and Slee (2016) is also a first introduction to the underlying data that we’ll be working with over the rest of term. D’Ignazio and Klein (2020b) should be getting you thinking about how ‘cleaning’ is not just about hygiene but has real implications for what you can and will find in your data: data is often a lot less ‘tidy’ than tidyr might lead you think! You should almost never be claiming that your (social) data represents the ‘universe’ of behaviours or is somehow ‘complete’.
Pre-Recorded Lectures
Come to class having watched:
| Session | Video | Presentation |
|---|---|---|
| Logic | Video | Slides |
| Randomness | Video | Slides |
| Data | Video | Slides |
| Pandas | Video | Slides |
Practical
In this practical we will begin working with the InsideAirbnb data, which you will have briefly examined in CASA0005. This week we focus on the first ‘class’ of data in the data set: simple numeric columns. We will see how to use Pandas for (simple) visualisation and (the beginnings of) analysis. It is hoped that you will see how Pandas combines and builds on techniques that we’ve already seen: while Pandas is incredibly sophisticated, the underlying concepts have been covered in the preceding three weeks! At this point we will also begin to make use of Pandas functionality to subset and explore the data.
The practical focusses on:
- Seeing how Pandas is ‘just’ a sophisticated extension of what we’ve already done.
- Familiarising yourself with Pandas functionality.
- Performing basic data cleaning and exploration tasks (including visualisation).
- Selecting and aggregating data in pandas.
To access the practical: