On Writing & Coding

Jon Reades - j.reades@ucl.ac.uk

1st October 2025

Writing & Coding == Thinking

If writing down your ideas always makes them more precise and more complete, then no one who hasn’t written about a topic has fully formed ideas about it. And someone who never writes has no fully formed ideas about anything nontrivial. ~ Graham (2022)

 

Writing is thinking. To write well is to think clearly. That’s why it’s so hard. ~ McCullough (2002)

A story of struggle. Frustration. Discovery. Learning. But you need to tell that story the right way.

Many of you will have learned some ‘system’ for writing in school. The inverted pyramid or something like that. In university, in my literary theory class I picked up the pyramid approach: taking a single sentence and unpacking that into the themes of the entire book.

There are mystery novels. Romance novels. Economist articles. Teen Vogue articles. They are all telling stories. They all do this in different ways.

There are no shortcuts to thinking and learning.

Large Language Models (LLMs)

ChatGPT from OpenAI (an increasingly ‘ironic’ name) is simply the most famous of a growing number of Large Language Models that draw on information found on the web and in open texts to perform sophisticated summarisation tasks.

Why Use it for Coding?

Many programmers use LLMs in coding for three reasons:

1. They can help to spot bugs, redundancy, and other issues that impact the performance of large applications (i.e. feedback).
2. They can provide information about different libraries and strategies the developer can use, as well as completing code begun by the developer (i.e. guidance or training).
3. They can help to ‘translate’ code and design patterns between languages (i.e. re-use).

These accelerate code production, but there are significant doubts about code quality.

Why Use it to Learn to Code?

Many students use LLMs to write their code for three reasons:

1. They are available at all hours as a personal tutor and advisor.
2. They write better code, more quickly than a learner.
3. They can employ advanced techniques right away.

Pro-Tip

Only one of these is a good reason.

Why Use It for Writing?

Many writers use LLMs for three reasons:

1. They can help to ensure consistency and suggest changes in tone or word choice (i.e. feedback).
2. They can provide information about different strategies the writer can use to tackle a problem (i.e. guidance or training).
3. They can help to ‘translate’ text between languages (i.e. re-use).

These accelerate word production, but there are significant doubts about quality.

Why Use it to Learn to Write (Academic English)?

Many students use LLMs to write their documents for three reasons:

1. They are available at all hours as a personal tutor and advisor.
2. They write better sentences, more quickly than a learner.
3. They draw on a wider range of sources and styles than a learner.

Pro-Tip

Only one of these is a good reason.

Don’t Take My Word for It

Using AI in education is like using a forklift at the gym. The weights do not actually need to be moved from place to place. That is not the work. The work is what happens within you. ~ Seemingly a conversation on YouTube?

Excellent! I’ve always heard students say, “When will I ever use this in real life?” And no teacher ever said, “These classes are not the work. The work is what happens within you.” Critical thinking, problem solving, the idea that the world is knowable. That is the work happening with you.

The problem is that many students don’t come to uni to learn to write essays, even those that come to learn (not just ‘get a degree’) often see learning as an accumulation of knowledge rather than the strengthening of a skill. So this gym analogy is really helpful for that too

As well…

From Richmond (2025):

Research has long shown that meaningfully processing new pieces of information – in other words, thinking hard about something – is vital for retaining that information in your long-term memory… if GenAI tools are used uncritically, there is a risk that they could disrupt or even circumvent the acquisition of new knowledge and the enhancement of existing knowledge – leading to poorer outcomes for learners of all ages.

Moreover, the impact of AI tools appears to differ between groups of students. Younger students and those with lower prior attainment were more likely to use it in ways that are detrimental to their learning (for example, copying and pasting work without reflection). Research has also shown similar effects of GenAI tools among workers, not just students.

Danger, Will Robinson!

Here’s what we’ve noticed about LLM use so far:

• Over-use of flowery language/complex code.
• Over-confidence in recommending solutions.
• Lack of overall structure/coherence.
• Use of non-existent or irrelevant references/coding libraries.¹

The Underlying Issue

• Lack of actual learning by the student.
• Undermining of learning by other students.
• Pointless work by staff.
• Deluge of bullshit (Bergstrom and West 2025)

This is very much a ‘brave new world’ and we are all trying to figure it out on the fly.

• We think that ‘intricate methodologies’ and ‘exhaustive reviews’ aren’t just overblown, they actually invite you to be marked down because youre methodology or review are not intricate or exhaustive.
• We find individual paragraphs that seem reasonable but the whole doesn’t ‘work’ as a single output.
• One of our PhD students has created a tool to help us find the papers in your bibliography that don’t actually exist, or that seem superficially useful but are not relevant in practice.
• Fundamentally, if you’re substituting the LLM for thinking then you’re not going to learn and that means that you are not going to be able to draw the wider connections that mark out strong submissions on assessments and the kind of person that we, or any company, might want to hire!

Basically, they are overconfident and, as a result, tend to either: a) lead the student to think it’s not worth the effort of learning (leading the student into deep trouble); or b) lead the student to overconfidence in their self-assessment of submissions or skills (leading the student into deep trouble).

1. This has even fueled security vulnerabilities in code using ‘slopsquatting’!

We Recommend…

LLMs like ChatGPT can help you to learn to be a better coder or writer by providing guidance and feedback, but for many applications a competent human being will be faster and have a better grasp of the purpose and rationale.

LLMs as co-authors

Using ChatGPT as your co-pilot is not the same as using ChatGPT as your co-author. In this module the latter is still considered plagiarism.

The people making the best use of LLMs are people who already know how to code or write.

Additional Resources

• The Myth of the ‘Genius Programmer’ (by Google Devs)
• Program-Aided Language Models
• Chain of Thought Prompting
• ChatGPT is a blurry JPEG of the Internet ¹
• Why Meta’s latest large language model survived only three days online ²
• Modern-Day Oracles or Bullshit Machines?
• Why We’re Not Using AI in This Course, Despite Its Obvious Benefits

• ‘How can I know what I think till I see what I say?’: How AI is changing education and writing
• What does it mean if students think that AI is more intelligent than they are?
• A Student’s Guide to Not Writing with ChatGPT
• A nice tutorial on distinguishing between plagiarism and paraphrasing.
• Greene, A. E. (2013). Writing science in plain English. University of Chicago Press.³
• Sword, H. (2017). Air & light & time & space: How successful academics write. Harvard University Press.⁴
• LLMs can’t stop making up software dependencies and sabotaging everything

1. Probably the best ‘lay person’s’ explanation of how LLMs work/fall apart you’ll ever read.

2. And this one was trained on scientific articles!

3. Not seemingly available for free, but I found a nice little summary (with typos) here.

4. Available for free via JStor.

Thank You

References

Bergstrom, C. T., and J. D. West. 2025. “Modern-Day Oracles or Bullshit Machines.” https://thebullshitmachines.com.

Goel, N. 2025. “New Junior Developers Can’t Actually Code.” https://nmn.gl/blog/ai-and-learning.

Graham, P. 2022. “Putting Ideas into Words.” 2022. https://paulgraham.com/words.html.

Klaas, B. 2025. “The Death of the Student Essay–and the Future of Cognition.” https://www.forkingpaths.co/p/the-death-of-the-student-essayand.

Lund, C. 2025. “The Incuriosity Engine.” https://cooperlund.medium.com/the-incuriosity-engine-16bdf41e229d.

McCullough, D. 2002. “Interview with NEH chairman Bruce Cole.” Humanities Magazine.

Richmond, T. 2025. “EducAItion, educAItion, educAItion: Could Generative Artificial Intelligence pose a risk to educational standards?” Social Market Foundation. 2025. https://www.smf.co.uk/publications/ai-and-learning/.