Are You Just
‘Organising It’?

Part 2

Jon Reades - j.reades@ucl.ac.uk

22nd October 2026

What’s the Key (Idea)?

Reports or Events?

stories = [('BBC','Chibok'), ('CNN','Chibok'),
           ('Japan Times','Chibok'), ('SaharaReporters','Chibok')]

counts = {}
for outlet, event in stories:
  counts[event] = counts.get(event, 0) + 1

print(len(stories))
print(counts)

4
{'Chibok': 4}

Four kidnappings, or one kidnapping reported four times?

Illustrative data, not GDELT’s: these are named on p.2 of the Data Feminism PDF.

Same data, two keys. Count by story and you get 4; count by event and you get 1. Remember from Dictionaries: “the key is unique. Always.” The choice of key is the choice of what you’re counting.

Then the next slide.

What is GDELT Measuring?

So what is it really that GDELT is measuring? What events are happening in the world, or what the major international news organizations are focusing their attention on? ~ D’Ignazio and Klein (2020, 16)

Neighbours, 1 minute: which key would answer each of those two questions?

Spot the Problem

From LOLs (“Making This Useful”):

my_cities = [
  ['London', 51.5072, 0.1275, +0],
  ['New York', 40.7127, 74.0059, -5],
  ['Tokyo', 35.6833, 139.6833, +8]
]

What’s wrong with this data?

Neighbours, 1 minute. Shown exactly as it appears in the video, unchanged.

Hint if the room is stuck: “Which side of Greenwich is London? Which side is New York?”

The Numbers Don’t Speak for Themselves

	On my slide	Actually
London longitude	0.1275	-0.1275
New York longitude	74.0059	-74.0059
Tokyo time zone1	+8	+9 (winter), +8 (summer)
New York time zone	-5	-5 (winter), -4 (briefly in spring)

Actually, it depends on where you’re measuring from: London or UTC!

“‘Raw data’ is an oxymoron” ~ Gitelman and Jackson, quoted in D’Ignazio and Klein (2020, 13)

These numbers have been in my slides for years, but numbers on a slide look authoritative. Those who did spot them had context: they know where New York is. In this case, the structure isn’t wrong: it’s incomplete. The structure has nowhere to say “when”.

In ds2 the keys are the only metadata, so a Python dictionary’s keys are a (very thin) data dictionary. Is tz in hours? Is it summer time?

D’Ignazio and Klein (p.6): “If you are lucky you might get […] a data dictionary that describes what each column […] means.” My margin note on p.6 (“Most data arrive on our computational doorstep context-free”).

Discuss the problems I’ve had with EPC data (≤ 60 seconds).

1. Claude and I misunderstood each other multiple times

What Can’t Be a Column?

What place-based “information” doesn’t fit on a shelf or in a database? What are the non-textual, un-recordable forms of cultural memory? ~ Mattern (2017)

Follow-up quote if needed: “We, humans, make urban information by various means: through sensory experience, through long-term exposure to a place, and, yes, by systematically filtering data.”

Neighbours, 2 minutes: name one thing about a city you know that can’t be a key or a column in ds2. Take three answers, from pairs in different parts of the room.

My margin note (printout p.4): “What does the archive/data set not capture?”

Then the week’s question again: was choosing ds2 just organising?

Break It, Then Fix It

Why the Command Line?

The last 100,000 crimes reported to Chicago police in 2025 (28MB)¹:

wc -l data/crimes2025.csv                 # how many rows?
head -2 data/crimes2025.csv               # what are the columns?
cut -d, -f6 data/crimes2025.csv | sort | uniq -c | sort -rn | head

About two seconds for 100,000 rows, and no Python.

Download it beforehand. It took about 40 seconds on home broadband (16 Oct). Show the command, but don’t run it on the room’s wifi. The server sometimes returns an empty file with no error (it happened once on 16 Oct): check with wc -l and run it again if you get 0.

curl -L -o data/crimes2025.csv 'https://data.cityofchicago.org/resource/t7ek-mgzi.csv?$limit=100000&$order=date%20DESC'

This is the city’s API, as in the Python slide, with a limit and a fixed order (latest first: 30 July to 31 December 2025). Without $limit the API stops at 1,000 rows; with it set to 300,000 you get the whole year (238,150 rows, 69MB, about a minute). Don’t use the portal’s “Export” link (/api/views/.../rows.csv): it’s deprecated and the server says it stops on 22 December 2026.

This is the “Crimes - 2025” extract (t7ek-mgzi). The 2022 one the course used before is 9hwr-2zxp. The full “Crimes - 2001 to Present” (ijzp-q8t2) is too big to demo.

As of 22 Oct: top types THEFT 22,985, BATTERY 18,038, CRIMINAL DAMAGE 11,394. The API puts every field in quotes, so the output reads "THEFT". Records are revised daily, so re-run before class.

Why not Excel? 100,000 rows will open (Excel’s limit is 1,048,576; the 2001-to-present file is well over that). The real problem is what Excel does to the values: 66,222 of these rows have an IUCR crime code with a leading zero ("0560"), and Excel turns it into the number 560. beat ("0931") and district ("009") go the same way. Excel decides what the data mean as it opens the file, which is today’s question.

This is the pay-off for week 2’s CLI video: | chains small tools. The same pipe idea is behind git log | grep.

1. Downloaded via curl -L -o data/crimes2025.csv 'https://data.cityofchicago.org/resource/t7ek-mgzi.csv?$limit=100000&$order=date%20DESC'

Is That Right?

grep -c '"HOMICIDE"' data/crimes2025.csv                    # 194
grep '"HOMICIDE"' data/crimes2025.csv | grep -c '"true"'    # arrests?

That’s not the number of homicides with an arrest: it’s 3 too many. Why?

awk -F, '{print NF}' data/crimes2025.csv | sort | uniq -c   # 22 columns?

Ask the room (pairs, 1 minute) before revealing: “Is 83 the number of homicides with an arrest?”

Why it’s wrong: arrest and domestic are both "true"/"false" columns next to each other. grep '"true"' also matches the 3 homicides with no arrest that were domestic.

The awk line: most rows have 22 fields, but about 1,800 have 23–25 (1,814 on 22 Oct), because descriptions like “AGG. DOMESTIC BATTERY - HANDS, FISTS, FEET, SERIOUS INJURY” contain commas inside quotes. cut -d, doesn’t know about quotes. Column 6 happens to come before the problem, so the first count was right, but only by luck.

Back to the week’s question: the storage format (CSV, and someone’s choice of description text) decides what a quick tool can see. The CLI is fast; it isn’t clever.

Two more data biography points if time:

• updated_on: about 193,000 of the 238,150 rows for 2025 were revised in 2026. A ‘record’ is not fixed when it’s written.
• The city says the data shouldn’t be used to compare over time, because classifications change after preliminary reporting. These are reports, as with GDELT.

From CSV to a Dict of Lists

import csv
import urllib.request

url = 'https://data.cityofchicago.org/resource/t7ek-mgzi.csv?$limit=20'

with urllib.request.urlopen(url) as f:
    rows = list(csv.reader(f.read().decode('utf-8').splitlines()))

header = rows[0]   # the column names
data   = rows[1:]  # a list of lists: one list per crime

crimes = {}        # a dict of lists: one list per column
for col in ['date', 'primary_type', 'block', 'arrest', 'latitude']:
    i = header.index(col)
    crimes[col] = []
    for row in data:
        crimes[col].append(row[i])

Copy and modify: this runs as-is in the SDS container, with no download. The ?$limit=20 asks the city’s API for just 20 rows. (The API’s column names are lower case with underscores, unlike the CSV export: primary_type, not Primary Type.)

We’ll see how this code works across later weeks. Focus on how the 3.2 list of lists becomes the 3.3 dict of lists. header.index(col) is the same trick as ds2['name'].index('Tokyo').

Invite them to change the column list, or $limit=20 to $limit=100.

Ask the Data

In pairs, write down the code before I run it:

1. How many of the 20 led to an arrest?
2. What is the average latitude?
3. Look at rows 2 and 4. One crime or two?

Switch to JupyterLab in the container. Ask a pair for the code for each one, type what they say, and run it (wrong code is fine: the error is the lesson).

Answers:

1. crimes['arrest'].count('true') → 4. Trap: the values are the strings 'true' and 'false', not True/False, so crimes['arrest'].count(True) gives 0.
2. sum(crimes['latitude']) → TypeError: everything is a string. We have to decide it’s a number: float(...) in a loop. Back to the week’s question: the CSV doesn’t store meaning, so you have to supply it.
3. Both are PUBLIC PEACE VIOLATION, “OTHER VIOLATION”, on an AIRCRAFT at 100XX W OHARE ST, at 23:54 on New Year’s Eve, with different case numbers (JK100014, JK100011). Two reports of one incident, or two incidents? You can’t tell from the data. That’s GDELT again. (Rows 9 and 10 are similar: a theft and a credit card fraud at the same place and time.)

Cut for time, but use if you’re ahead: “What was the first crime, and where?” (crimes['primary_type'][0], crimes['block'][0] → ASSAULT, 050XX S PAULINA ST) and “Where was the first THEFT?” (crimes['block'][crimes['primary_type'].index('THEFT')] → 006XX E GRAND AVE, row 9: the “time zone of Tokyo” pattern from the DOLs video).

Gitting More
Out of Life

Break It

A reproducible sequence of interactions:

mkdir tmp
cd tmp
git init
echo "Test" > fileA.md
git add fileA.md
git commit -m "Added."
git rm fileA.md
git commit -m "Removed file."
ls                       # fileA.md has gone
git log --oneline        # find the hash before the removal
git checkout <hash>
ls                       # fileA.md is back

Live, in the SDS container: the Recovery slide from the Git video, done for real. Set up the repo beforehand (README → fileA.md → commit, as in the video).

Ask before running git checkout: “What will git status say now?”

Detached HEAD

git status               # HEAD detached at <hash>
git switch -             # back to the branch you were on

To bring one file back into the present:

git restore --source=<hash> fileA.md
git add fileA.md
git commit -m "Restored fileA.md"

This is the “getting back to HEAD” discussion that the Git video promises but never delivers.

git switch - works whether your branch is main or master. GitHub now defaults to main, but the video slides say master.

Commits on a Plane

	CVS (1990), Subversion (2000)	Git (2005)
Where is the history?	On one central server	In every copy
Commit offline?	No	Yes
GitHub is…	(n/a)	just another copy

So commit and push are different things.

The point the video doesn’t make: Git is peer-to-peer. Every clone is a complete repository, with the full history. GitHub is a copy that everyone has agreed to treat as the main one; Git itself doesn’t know that.

Can be brief, the history is optional. The history: CVS and then Subversion (“CVS done right”) were centralised: you needed the server to commit. The Linux kernel moved to BitKeeper, a proprietary distributed system, in 2002. In April 2005 the free licence was withdrawn after a dispute over reverse-engineering its protocol, and Linus Torvalds wrote Git in a matter of days to replace it. (Mercurial started that same month for the same reason.)

Why it’s weird: there’s no single ‘true’ copy, so pull and push are needed to make copies agree. Why it’s useful: you can commit on a plane, and the history survives if GitHub disappears. See: the great Moodle/PHP exploit of 21 October 2026!

This sets up the next slide: a commit on your laptop is not a backup until you push.

Pull, Then Push

git pull                 # get any changes from GitHub first
git add <notebook.ipynb>
git commit -m "What I did, and why"
git push                 # only now are you backed up

When asked for a password, GitHub wants a personal access token (or an SSH key), not your account password.

The video’s workflow slide leaves out pull, but the learning outcome includes it.

Why the command line rather than a GUI: it shows what’s happening behind the scenes. add and commit are separate steps, and a GUI usually hides that. The separation is the point: you choose what goes into the record.

Everyone should have a GitHub account by now. The first push from the command line is where the password prompt catches people out.

Link back to 3.6: in the group project you’ll be pulling and pushing to one repository with up to four other people, and the commit history is how we see who did what if a group falls out.

Spare question if time: “You commit, and then your laptop is stolen. Is your work on GitHub?” (No: all changes are local until pushed.) Story: the flatmate’s laptop.

A Commit Message Is Context

The way through this predicament is by considering context, a process that includes understanding the provenance and environment from which the data was collected […] ~ D’Ignazio and Klein (2020, 27)

git log is the provenance of your work. Write the message for the person who will read it in six months.

D’Ignazio and Klein’s “data biography” (p.24) asks five questions: Where did it come from? Who collected it? When? How was it collected? Why was it collected? A good commit history answers most of them for your own code. My margin note: “Something to think about for the assessment, perhaps?”

Bridge to the practical: the final commit of the notebook. Ask for a message that would make sense to a stranger.

Intro to the practical

Here’s what we’re covering:

• Bedding in your Git/GitHub workflow. We’re trying to help you remember via repetition here and get good habits started early.
• Lists and dictionaries are the ‘workhorse’ data structures of Python: you’ll use them all the time even once you know of other, more powerful ways to organise and manage data. You might, for instance, have a list of advanced data structures that you want to iterate over. Or a dictionary of parameters that you want to test in a model.
• We’re also going to stretch your understanding of how these two data structures interrelate and, crucially, look at the DOL, which is analogous to how pandas — probably the single biggest reason Python dominates data science — manages and represents data to you.

Before You Go…

Complete the sentence:

 

How I store data is / isn’t a neutral choice, because…

 

A selection will open next week’s session.

Input

Provide your input here: form.menti.com/s/albxtzgfabbm

Mentimeter QR Code

Mentimeter (open text) in the second browser tab. Week 2 had 17 responses from about 110 students, even though we finished early, so time wasn’t the problem.

Start it at a fixed point (minute 110, or earlier if we’re ahead), leave the code on screen, and give 3 minutes of silence before saying anything else. Repeat that it’s anonymous and that today opened with three of last week’s. Week 4 opens with a selection of these (Jon isn’t teaching week 4, so send a selection to the lecturer covering it).

Thank You

References

D’Ignazio, Catherine, and Lauren F. Klein. 2020. “Data Feminism.” In. MIT Press. https://data-feminism.mitpress.mit.edu/.

Mattern, Shannon. 2017. “A City Is Not a Computer.” Places Journal. https://doi.org/10.22269/170207.