This essay was edited out of a chapter of my book, The Intelligence Illusion: a practical guide to the business risks of Generative AI, with minor alterations.

“See the choice of dreams”, and then worry about it

AI software development – the “what is this for?” part – has never had much of a unifying vision. AI research, sure, they have a vision: they want intelligent machines first, figure out what to do with them second.

They dream of a robot future.

Some parts of the research monomania end up having clear software benefits. Being able to point a computer at an image and have it get at least a rough idea of what the picture is of is neat and that came from Machine Learning research. It didn’t come with a single specific “what is this for?” vision except, you know, “how is our robot going to see?”, but it made up for it by being obviously useful in a general sort of way. It’s a capability that’s now built into pretty much all of our devices. As a feature that’s now integrated into our lives, it’s a microcosm of the issues we have with innovations coming out of AI research:

1. It has helped the blind and partially-sighted access places and media they could not before. A genuine technological miracle.
2. It lets our photo apps automatically find all the pictures of Grandpa using facial recognition.
3. It has become one of the basic building blocks of an authoritarian police state, given multinational corporations the surveillance power that previously only existed in dystopian nightmares, and extended pervasive digital surveillance into our physical lives, making all of our lives less free and less safe.

One of these benefits is not like the other.

Universal facial recognition is terrifying when it works perfectly and a nightmare when it’s flawed. It exaggerates power imbalances and disproportionally enables bad actors and authoritarians. It’s equal parts pleasant domestic miracles and blighted social and political horror. Generative AI is likely to follow the same path.

In the absence of a unifying vision, the tech industry simply does what makes the most money for people working in the tech industry – greed fills the void where there should have been vision. Companies such as Amazon didn’t hesitate to sell facial recognition services to law enforcement – until the backlash forced them to stop.^[2]

This might just be capitalism, but the ‘just’ in that phrase feels quite different when the industry in question is peddling synthetic miracles.

Greed might be inevitable. It might always seep into the cracks, break apart the concrete foundations our ivory towers are built on. But, having a coherent unifying vision that’s backed by clear values does a remarkable job of holding off the decay.

Even today, the web is like living fossil, a preserved relic from a different era. Anybody can put up a website. Anybody can run a business over it. I can build an app or service, send the URL to anybody I like, and most people in the world will be able to run it without asking anybody’s permission. There are rules you have to follow, obviously, but those are remarkably straightforward if you aren’t actively spying on people or messing around with their data – especially when you’re working on a comparatively small scale.

You can trace the lineage of the vision behind the web from Tim Berners-Lee^[3] in 1989, through Ted Nelson in 1974 and Douglas Engelbart in 1968, all the way to Vannevar Bush’s article As We May Think in the Atlantic Monthly back in 1945.^[4]

All of their books, software prototypes, theories, and ideas run along the single continuing thread of the hypertext concept – links – as a new kind of punctuation mark^[5] that connects the information of the world together in a coherent network. It’s a vision that encompasses concept, functionality, interface, and values – and it persists to this day, despite decades of greed, abuse, surveillance, and shitty ads. It’s a unifying vision of a world that’s simultaneously technological and literate. This vision is part of what has kept it alive. Despite the frustrations, pain, and the flaws, working on making parts of the web is a privilege.

While AI researchers are busy trying to build their robot dream, generative AI software has no such unifying vision. Some vendors are bent on replacing humans at their jobs – effectively promoting their software as “AI” illustrators, voice actors,^[6] even “robot” lawyers.^[7] – and then look surprised at sheer enormity of the anger that they get in response. Other vendors are resurrecting Microsoft’s 1990s dream of intelligent agents, assistants, or copilots who operate in the context of the software you use – extending Clippy’s lineage into the modern world.^[8] The rest seem to have lazily reached for the first interface metaphor they could think of – the chatbot – with no thought or even the vaguest idea of how it should actually integrate with the rest of the work we’re doing.

As much as it makes your average MBA salivate, “let’s replace people with something shittier but cheaper” isn’t much of a vision for software development and user interface design, which leaves those of us who are genuinely curious about the applications of the technology with the other two paths.

Those seem to be converging towards a single idea: “Human-in-the-loop”. It’s the idea that in the interactive loop with the AI software, the decision-making, choices, and actions are made by the human.^[9] Instead of full automation there’s feedback from the AI as the tasks progress and the human responds to that by interacting with the various user interface affordances provided by the system.

In other words, the human sits in front of software, uses it as they would any other software, and then it does stuff for them as any other software does, except in an AI way.

The grand unifying vision of AI-assisted software is that you should use it to make software.

That’s an idea that’s only remarkable because of how many AI enthusiasts think they can do away with the people part of getting things done.

Acquiesce, or mitigating the inevitable

In an earlier chapter I wrote about the failure of an AI model designed to predict the onset of sepsis, how external reviewers discovered the flaws, which then led to the vendor updating and improving it.

At one hospital, UC Health in Colorado, they found that the system still wasn’t that useful: “the ratio of false alarms to true positives was about 30 to 1”.^[10]

To salvage their investment UC Health changed their approach with the system. Instead of using the AI as an autonomous prediction system that sent out alerts to overworked doctors and nurses, they put together a special monitoring team of clinicians that used live video feeds to helped filter out the false alarms. That team built relationships with bedside nurses throughout the hospital. Where the AI system alone was utterly useless, the human team, assisted by their relationships with the nurses and the AI system, was estimated to save about 211 lives annually.

AI on its own was worse than nothing while AI as an assistant saved lives, a clear demonstration of the value of human-in-the-loop. It’s a heart-warming parable that lends credence to the ambitions of those who are trying to make “AI assistants” happen.

This, and other stories like it, are going to be the foundation myths of a thousand new AI services – the seeds of a new computing revolution.

Or, more specifically, it’s a certain kind of fertiliser. The kind that smells.

Case studies are amazing tools. You can pick one instance where everything worked out – an exercise absent of disaster – has a nice “all is lost” moment that gets turned around, and throw together a just-so story that proves exactly the point you want. There isn’t anything anybody can do to disprove it without particle-colliding themselves into an alternate reality. There is no way to ‘science’ a case study unless you have access to a parallel universe as a control. They’re all just stories that short-circuit our thinking.

For every UC Health sepsis story there are a hundred systems that didn’t work. Even the UC Health story itself is dubious once you dig into it. Was it the AI that saved 211 lives? Or was it having a specialised team of clinicians watching all the at-risk patients around the clock, using live feeds? Or, was it the relationships the clinicians developed with the bedside nurses? If you’d put together that same team with the same infrastructure, but using a simpler, cheaper algorithm based on vital sign monitors, would that have done the same job? Why didn’t UC Health try that first – simpler, cheaper, faster to set up – if what they wanted was to save lives?

The answer is simple: they’d already bought a broken AI system. They did what they had to in terms of making sure their investment did eventually save lives, but it leaves us with this unanswered question: if they had spent that same amount of money on building teams and a system for detecting sepsis, but without AI, would it have worked better or worse?

We can’t know, and that’s why case studies are a favoured tool by MBAs, startups, and consultants all over the world. You can just pick a story that proves what you want and ignore the other hundred that don’t.

Once Generative AI becomes a broad movement in software, facts and science won’t matter, and the stories will take over. Ted Nelson was writing about computers and programming in a more general way and in a different era, but he’s right here too: the stories about AI software aren’t scientific and trying to make them look scientific is an outrage.

That’s what the AI software vendors are doing with their marketing performances that look like scientific papers, the ‘studies’ that are little more than sales exercises, and the entirety of their rhetoric about being on the verge of AGI – how we need to make sure those future robot gods are our slaves and not overlords.

It’s storytelling.

Fighting that with another ‘science’ performance is futile. In a war of theatrics, the act with the biggest budget wins the crowd. We can chip away at the foundations with peer-reviewed papers and research that show flaws and failures, but ultimately what will decide this in the decades to come is the software – how well it’s designed, how effective, how productive, and the long-term failures and successes in real workplaces.

That’s where the three core flaws of the assistant model is going to be a problem.

I mentioned two of them earlier, automation and anchoring biases. We, as human beings, have a strong tendency to trust machines over our own judgement.^[11] This kills people, as it’s been a major problem in aviation.^[12] Anchoring bias comes from our tendency to let the initial perceptions, thoughts, and ideas set the context for everything that follows. AI adds a third issue: anthropomorphism.^[13] Even the smartest people you know will fall for this effect as large language models are incredibly convincing.^[14] These biases combined lead people to feel even more confident in the AI’s work and believe that it’s done a better job than it has.^[15]

We’re using the AI tools for cognitive assistance. This means that we are specifically using them to think less.^[16] In every other industry this dynamic inevitably triggers our automation bias and compromises our judgement of the work done by the tools.^[17] We use the assistant to think less, so we do.

These models are incredibly fluent and – as we saw at the start of this book – are consistently presented by their vendors as near-AGI. This triggers our instinct towards anthropomorphism,^[18] making us feel like we have a fully human-level intelligence assisting us, creating an intelligence illusion that again hinders are ability to properly assess the work it’s doing for us.^[19]

Anthropomorphism, when applied to AI chatbots has been called the “Eliza effect”. It was first observed by Joseph Weizenbaum when he saw how people responded to and interacted with the comparatively primitive ‘AI’ chatbot, Eliza, that he created back in 1966.

Fluent AI models create an anthropomorphism effect that sways even those who knew that the AI was nothing more than a simplistic program, even by 1966 standards.^[21]

The intelligence illusion, the conviction that these are artificial minds capable of powerful reasoning, when combined with anthropomorphism supercharges our automation bias. Our first response to even the most inane pablum from a language model chatbot is awe and wonder. It sounds like a real person at your beck and call! The drive to treat it as, not just a person, but an expert is irresistible. For most people, the incoherence, mediocrity, hallucinations, plagiarism, and biases won’t register over their sense of wonder.

This anthropomorphism-induced delusion is the fatal flaw of all AI assistant and copilot systems. It all but guarantees that – even though the outcome you get from using them is likely to be worse than if you’d done it yourself, because of the flaws inherent in these models – you will feel more confident in it, not less.

Every human-in-the-loop and assistant-style AI system I’ve seen has these defects. Some of them even do their best to exacerbate them by making the assistant adopt a confident tone or an affable demeanour.

Those who use these AI systems are likely to get worse results and still be more confident in the resulting output than they would have in their own. Their work will suffer, but they will feel like it has improved. This is a recipe for fanatical evangelism and incredible revenue growth. It all but guarantees that we’ll see a financial bubble of some kind around AI. The only question is its size and duration. The more effective the Generative AI systems are, the bigger the bubble. The less effective they are, the faster it’ll pop.

We’ll probably get some good software out of it – especially when it comes to converting or modifying text and media – but it’s the nature of bubbles to create crap. A software bubble is the flowering of a thousand first-movers – countless startups and tech companies, most of them utterly clueless about what they’re working with, building the first bad iterations of what they hope is a good idea. We don’t know yet what the ideal, productive AI-assisted productivity software will look like, but we do know that we’re unlikely to see many examples of it in the first generation.

Meanwhile, the tech industry will dream of exponential growth.

The roads home

I have lived abroad for most of the past twenty years. The web let me work wherever I wanted without losing touch with my friends and family. The tools the web offers gave me freedom that I couldn’t have imagined when I was a child.

This worked well for a while. I’ve had the joy of living in a number of wonderful cities and amazing neighbourhoods and communities.

I grew older and, with age, those around you also grow older. Some of them get sicker. A video chat doesn’t fill the void you feel when somebody you care about is lying in a hospital bed. But the freedom the web provides works in the other direction as well. I could live near those I care about and the web meant I could keep doing my job no matter where I was.

I decided to move back home to Iceland. As I was preparing my move, the COVID-19 pandemic struck, and the rest of the world discovered what I had known for decades: the web abstracts distance. You can work where you want. I made my way home, despite the collapse of international airline travel. From Montréal to Toronto. Toronto to Amsterdam. Finally, I flew from Amsterdam to Iceland.

Back in Iceland, I settled in Hveragerði, a small town of about 2700 people in the south of Iceland. Keeping with theme of my realisation that the web and related technologies meant that location mattered less, I could pick a place that suited my personal needs. It’s a nice town. The weather here can be interesting – this is Iceland after all – which often leads to road closures in the winter. But there are three separate roads that connect this region with the capital, so even though a couple of the roads get closed due to snow or ice there’s always the third. Because we know what to expect from the weather, most regions in Iceland invest in their infrastructure. We try to make sure we can keep everything going even when a bad storm hits us. There are redundancies and, for the most part, they work.

We can’t say the same about the software that we have today – that we use for our work. Even though many organisations have returned to the office, partially or fully, we are still using the same software that companies adopted for remote work. We use Google Docs, Zoom, Dropbox, or an equivalent competitor. Our files, documents, and processes are now tied to whatever app we’ve adopted.

If Google Docs goes down or has sporadic outages, then our work disappears with it. If our internet goes, the software blinks out. When the biggest data centre belonging to Amazon Web Services goes down, that breaks most of their services across all of their data centres, because they’re all interconnected, and almost all of our software breaks with it.^[22] Given our increasing reliance on centrally hosted software services, the impact of temporarily losing a data centre is severe, getting worse, and is now even happening because of the weather, caused by an increase in frequency and magnitude of heatwaves globally.^[23]

It doesn’t matter whether we ourselves are working remotely or in-office, all our software today is remote, and the connection to it can break in a thousand different ways.

There is only one road into town.

A global network means our software shouldn’t have to be centrally located in only a few specific buildings across the world. It should be spread throughout the network, on every device that’s connected to it. Our hardware devices shouldn’t have to be so reliant on the internet that core features cease to function just because they can’t phone home for a short while. Our information – public, private, professional – shouldn’t have to be controlled, collected, and stored by only a handful of corporations.

The software we have today is undermining the strongest advantage given to us by the internet: robust and distributed reliability. Our work depends on increasingly unreliable software. Their need to be always online means you feel every hiccup in your connection. Centralisation means that when something does go wrong, it’s potentially catastrophic as it affects everybody, everywhere, who is using that centralised app. This matters because things are going wrong, fast. There’s political unrest. Social instability. Cold wars. Hot wars. Trade wars. A climate crisis. Data that was just normal personal data one moment, becomes incriminating evidence a moment later when people’s rights are stripped away.

The software we have isn’t the software we need.

The opposite of good software

Modern software is remarkably fragile. We’ve gone from a software ecosystem that, a few years ago, was almost completely local, to one where everything is just cached – temporarily stored – at best. A decade ago what you worked with was on the computer itself. Your data was your own, and relatively safe if you kept decent care of your backup drive. The apps were yours, usually bought and paid for once – no subscription. Collaboration was always a bit tricky if you weren’t a software developer – we’ve always had somewhat decent collaborative tools in version control systems – but other people made do by using shared local servers or simply sending files over email.

This was a remarkably robust software ecosystem that tolerated all sorts of disasters, disconnections, and changes. We’ve dismantled it in less than a decade. Most of the apps we use for our work require an internet connection. Almost all of them are entirely cloud-based, where significant parts of the software runs on a server somewhere. Little of our work data is stored locally any more.

Generative AI serves to accelerate that trend.^[24] You needed 800 GB just to store GPT-3, without even running it.^[25] Later versions and ChatGPT are even bigger, running in parallel on multiple servers. The technology can be made to work locally, but that’s not where the hype is. The hype is for the already countless “AI for X” services who are all in the cloud and are all using services from OpenAI.^[26] Unless one of the big tech companies breaks ranks and builds into their Operating system a solid and tested large language model that has been ethically trained on documented data sets, what we’re going to get are agents and chatbots everywhere, each living in the cloud, fine-tuned for a task, and hooked up to whatever APIs the startups think are nifty. Why solve the hard problem of making a language model safer, better integrated into your software, and more sustainably developed, when you can hook up a finicky but flashy website with OpenAI and call it a startup?

The dream the tech industry chose is not science or progress. The dream they chose is that of easy money, because that’s the only dream the tech industry today is capable of seeing. Their vision is a mirage of craving.

Their want can only be met with another financial bubble, one that has to be more grand and world-changing than any other that preceded it. They crave the exponential to fulfil their dreams, but the only true exponential today’s twenty-something startup founder will experience is that of the escalating Climate Crisis. That won’t stop them from trying. Their hunger is likely to push them to ignore the social unrest and power shifts that AI systems cause.

The tech industry doesn’t just behave with your normal corporate greed. They want financial bubbles. They had a taste of the euphoria with the dot-com bubble and the hunger for it never went away.

The tech industry is also, as I argued at the start of this book, full of true believers in AI. Somebody who truly believes – sincerely believes that this will all be for the best – will push past the mass unemployment, organised disinformation, and wholesale deception. They will think that it will all be worth it. Once we get through the initial “disruption”, things will be better for everybody.

None of this is conducive to software design and development. It isn’t a mindset that leads you to do user research, observational studies, or usability experiments. It’s a drive that’s taking them away from what most people and their communities need. Where we need robust technology, they are giving us finicky AIs that misbehave at a badly worded sentence. Where we need privacy from both corporations and potentially hostile authorities, they push further and further into recording our lives. When we need software that works on the devices we have, for as long as they last, they give us software that only works on the latest and greatest. Sometimes, as with GPT-4, the software they make even requires systems so powerful that they only exist in a couple of locations on the planet.

But, don’t worry, they’ll sell us access – timeshare, really – but let’s call it “the cloud”. It only breaks some of the time.

Nothing they do is for us, even though it’s our money, our data, and our art, writing, and music they’re demanding. We aren’t customers to them – we’re just the people that pay. To tech companies, we are nothing more than a resource to be tapped. A number to be boosted to pump investor interest. They are not doing us any favours. What they want from us is simple: everything. All culture on their servers, made by their AI. All our work happening through them, assisted by their AI. The totality of our information, mediated by their AI. A vig collected on all existence.

One of the papers I’ve referred to a few times in this book is On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?^[27] It was the first paper to provide a cohesive and detailed overview of how large language models work, how they affect people, and the risks that they pose. This paper ultimately led to Google firing one of the key authors of the paper, Timnit Gebru, and forced other co-authors employed by Google to take their name off it.^[28] This continues to this day, where Google employees seem to be routinely discouraged from working on AI fairness or ethics.^[29]

When Microsoft launched Bing Chat, the first mainstream attempt to use a large language model as a front end for search – something that another co-author of On Stochastic Parrots, Emily M. Bender, had warned against in a separate paper titled Situating Search^[30] – this lead to the exact outcomes they had predicted. Strange behaviour^[31], threatening language^[32], falsehoods^[33] and lies^[34] ensued. Bing Chat played out exactly the way they expected.

Of course, Microsoft did the only rational thing it could when the risks of its products were revealed: it disbanded its AI ethics and safety team^[35] and rolled Bing Chat out to even more people.^[36] It now plans to push towards adding AI chatbots to everything, everywhere, no matter the cost.^[37]

Most of the tech organisations that had responsible AI or AI safety teams are disbanding them.^[38]

They seem to think it would be a mistake to worry about risks and problems – why worry about something you can probably fix?^[39] Who cares about the harm it does in the meantime?

Safe, for the tech industry, is too slow when you hunger for a bubble and want to ship more software, to more people, as fast as you can.

Designers of software user interfaces often imagine deliberately bad designs as an exercise – a way of demonstrating the principles of their craft by exploring their opposites. It’s a good way of demonstrating why a design principle matters, and it can provide tactile examples of who benefits from it and how.

If you asked me to imagine the software that would be the opposite of what we need as a society…

That app would look remarkably like ChatGPT.

The best way to support this newsletter or my blog is to buy one of my books, The Intelligence Illusion: a practical guide to the business risks of Generative AI or Out of the Software Crisis.

To be released on November 7, 2023. Pre-order now to get a $10 USD discount.

Software projects keep failing, not because we don’t have the right team or tools but because our software development system is broken. Out of the Software Crisis is a guide to fixing your software projects with systems-thinking making them more resilient to change and less likely to fail.

The print edition will be:

• Case bound with a glossy cover.
• 180 pages.
• Shipped from either the US, if you’re ordering from the US, or the UK if you’re ordering from anywhere else.
• Those ordering from outside the US or the UK might have to pay additional customs charges on delivery.
• The book should appear in regular book store distribution systems late November or early December.
• Print orders do not include the ebook as that would make VAT-handling and sales tax even more complicated.
• All pre-orders will get an automatic $10 discount automatically applied at checkout and get the book for only $50 USD.
• Payments are processed by PayPal. You should be able to finish the purchase as a “guest” without needing to create a PayPal account.

Praise for the ebook edition of Out of the Software Crisis: Systems-Thinking for Software Projects

The only issue so far with Baldur’s book is how distracted I am by the typography. It’s a bit too good 😅. – David Larlet

I’ve bought your book based solely on the extract about the business value of unit testing on the website. I read that and thought “YES! At last! Someone’s said it.” and wondered what else you have to say. I’m looking forward to reading it. :) – Chris Neale

I read Baldur’s book ‘Out of the Software Crisis’ on the flight over yesterday and I’d strongly recommend reading it. – Orta Therox

This is basically a book full of hot takes and I’m here for it. – James Harton

“Out of the Software Crisis” by Baldur is a great book and well worth the read. Highly quotable, and really challenged my view of software development in a few places. – Nolan Lawson

Really great as an introduction to systems thinking as applied to software design. – Jennifer Jiang

Love this quote about design from “Out of the software crisis” by Baldur: “Decoration [is] the least important thing design does for software…[aesthetics] is part of what design does. It isn’t what you hire designers to do.” – Jim Neilsen

This book “out of the software crisis” by Baldur Bjarnason is really good 👍. I cannot recommend it highly enough. – KimSia Sim

Just finished Baldur’s new book “Out of the Software Crisis”. It’s a whistle-stop tour of the history and current state of #software development, and an intro to systems thinking. Very relatable if you’re a #webdev, and has inspired me to read more… – Ben Walker

“Out of the Software Crisis” by Baldur is so good that I shared it with our executive management team. It’s short, non-technical, well-researched and identifies the key reasons software projects fail and how to improve the chance of success. Hoping this authoritative little book will help our team look beyond the waterfall/agile/safe/whatever dogmas to the importance of the wider organization in determining success. – @doctorlaura

For more information and to buy the ebook edition

• You can go to the pre-order page proper.
• See the main page for the book.
• Follow the newsletter.

This is hard when stock prices are largely driven by trends and pop culture and even harder when your own incentives are almost entirely driven by said trends. Often the decision is made above your pay grade and there’s little you can do to opt out short of braving the sketchiest job market for developers in years.

Opting out seems impossible and yet putting off adoption is the only reliable way of discovering the true impact of the technology.

Research is largely bought and paid for by vendors. The studies that aren’t and are attempting to critically study how these models work are hampered by poor access and outright hostility from the surrounding industry.

Even if we had a wealth of impartial research, coverage very definitely isn’t, at least in tech.

The sentiment towards generative models in non-tech media outlets has, in my opinion, shifted decidedly towards the negative, but that rarely affects decision-making in software development.

The problem is that software development is a complex system where it’s often hard to trace the effects of a change directly to its consequences. That’s why you need time.

Watch the companies that have gone all-in on generative models in software development.

• Will their productivity increase visibly over the next two or three years? Or will the productivity effects of generative models largely be a mirage?
• Will they release more features at a faster cadence?
• Will the quality of their software improve?
• Or will the number of serious defects increase?

That last part is vital. Because sometimes the groundbreaking and original new technology is just a bigger footgun with very little actual benefit. One of my worries with generative models in software development is that extensive cognitive automation and programming is an extremely bad fit as it makes us think less about what we’re doing, which makes us more likely to make careless mistakes.

We can drop Microsoft from this list. They have a long history of incompetent software development and poor security. Pick any year, any month of that year, and you will find a news story telling you about a major security issue in Microsoft code. They won’t be a good benchmark. Maybe, Github? They are independent enough to have a slightly different security culture.

We don’t know if Google actually lets their own developers use generative models on core products. Until that’s confirmed then they’re out.

A company like Sourcegraph should be on the list. A while back they hired Steve Yegge who is on the record for being exuberantly pro-“AI” in software development and their product roadmap gives the impression of a company that’s all-in.

One incident tells us nothing, but a series of them over the next few years would. If they and other “all-in” companies turn things around and then improve their productivity, that tells us something as well.

But it’s only time that will conclusively give us the answer if this tech is a groundbreaking new enhancement, or revolutionary own-foot-cannon.

I shouldn’t need to say this, but apparently I do: all existing generative model products are unethical.

I’m not talking about legality here. A thing can be legal but unethical. These models are built on people’s creative work, without their permission, and then integrated into products that directly threaten their livelihoods. Their work becomes a facet of the model’s output without attribution.

This is straightforwardly unethical, irrespective of the legality or how the models work internally. You are using people’s work to destroy their livelihoods. People should always come before software and models are software not people.

Once you include the many issues these models have with biases, privacy, and memorisation it becomes unambiguously clear that using this tech is harming people.

An ethical generative model product is possible, in theory, but none that are available today are. We have a few pseudo-open models that would qualify as ethical for research and study, but none that should be acceptable for a commercial product for widespread use.

All of my advice for mitigating the business risks of these products is for those who cannot opt out of using them. It’s for those of you who feel like you have to and would like to minimise the harm.

But if you’re voluntarily using these generative models without any outside force pressuring you to do so?

The simplest way to minimise the harm these models are doing and the risks they present is to just stop.

I would be very grateful.

One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.

But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and, if real, it would be completely unexplained.

LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.

LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.

There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.

There are two possible explanations for this effect:

1. The tech industry has accidentally invented the initial stages a completely new kind of mind, based on completely unknown principles, using completely unknown processes that have no parallel in the biological world.
2. The intelligence illusion is in the mind of the user and not in the LLM itself.

Many AI critics, including myself, are firmly in the second camp. It’s why I titled my book on the risks of generative “AI” The Intelligence Illusion.

For the past couple of months, I’ve been working on an idea that I think explains the mechanism of this intelligence illusion.

I now believe that there is even less intelligence and reasoning in these LLMs than I thought before.

Many of the proposed use cases now look like borderline fraudulent pseudoscience to me.

The rise of the mechanical psychic

The intelligence illusion seems to be based on the same mechanism as that of a psychic’s con, often called cold reading. It looks like an accidental automation of the same basic tactic.

By using validation statements, such as sentences that use the Forer effect, the chatbot and the psychic both give the impression of being able to make extremely specific answers, but those answers are in fact statistically generic.

The psychic uses these statements to give the impression of being able to read minds and hear the secrets of the dead.

The chatbot gives the impression of an intelligence that is specifically engaging with you and your work, but that impression is nothing more than a statistical trick.

This idea was first planted in my head when I was going over some of the statements people have been making about the reasoning of these “AI.”

I first thought that these were just classic cases of tech bubble enthusiasm, but no, “AI” has both taken a different crowd and the believers in the “AI” bubble sound very different from those of prior bubbles.

—“This is real. It’s a bit worrying, but it’s real.”

—“There really is something there. Not sure what to think of it, but I’ve experienced it myself.”

—“You need to keep your mind open to the possibilities. Once you do, you’ll see that there’s something to it.”

That’s when I remembered, triggered by a blog post by Terence Eden on the prevalence of Forer statements in chatbot replies. I have heard this before.

This specific blend of awe, disbelief, and dread all sound like the words of a victim of a mentalist scam artist—psychics.

The psychic’s con is a tried and true method for scamming people that has been honed through the ages.

What I describe below is one variation. There are many variations, but the core mechanism remains the same.

The Psychic’s Con

1. The Audience Selects Itself
Most people aren’t interested in psychics or the like, so the initial audience pool is already generally more open-minded and less critical than the population in general.

2. The Scene is Set
The initial audience is prepared. Lights are dimmed. The psychic is hyped up. Staff research the audience on social media or through conversation. The audience's demographics are noted.

3. Narrowing Down the Demographic
The psychic gauges the information they have on the audience, gestures towards a row or cluster, and makes a statement that sounds specific but is in fact statistically likely for the demographic. Usually at least one person reacts. If not, the psychic will imply that the secret is too embarrassing for the "real" person to come forward, reminds people that they're available for private readings, and tries again.

4. The Mark is Tested
The reaction indicates that the mark believes they were “read”. This leads to a burst of questions that, again, sound very specific but are actually statistically generic. If the mark doesn’t respond, the psychic declares the initial read a success and tries again.

5. The Subjective Validation Loop
The con begins in earnest. The psychic asks a series of questions that all sound very specific to the mark but are in reality just statistically probable guesses, based on their demographics and prior answers, phrased in a specific, highly confident way.

6. “Wow! That psychic is the real thing!”
The psychic ends the conversation and the mark is left with the sense that the psychic has uncanny powers. But the psychic isn’t the real thing. It’s all a con.

1. Audience selection

Seers, tarot card readers, psychics, mind readers aren’t all con artists. Sometimes the “psychic” is open about it all just being entertainment and aren’t pretending to be able to contact spirits or read minds. Some psychics do not have a profit motive at all, and without the grift it doesn’t seem fair to call somebody a con artist.

But many of them are con artists deliberately fooling people, and they all operate using the same basic mechanisms that begin well before the reading proper.

The audience is usually only composed of those already pre-disposed to believe in psychic phenomena and those they have managed to drag with them. Hardcore sceptics will almost always be in a very small minority of the audience, which both makes them easy to manage and provides social pressure on them to tone down their scepticism.

Those who attend are primed to believe and are already familiar with the mythology surrounding psychics. All of which helps them manage expectations and frame their performance.

2. Setting the scene

Usually the audience is reminded of the ground rules for how psychic readings “work” at the start of the performance. They are helped by the popularisation of these rules by media, cinema, and TV.

Everybody now “knows” that:

• Readings usually begin murky and unclear.
• They then become clearer as the “connection” to the “spirit world” gets stronger.
• Errors are expected. The “spirits” are often vague or hard to hear.
• Non-believers can weaken or even disrupt the connection.

Psychics also habitually research their audience, by mapping out their demographics, looking them up on social media, or even with informal interviews performed by staff mingling with attendees before the performance begins.

When the lights dim, the psychic should have a clear idea of which members of the audience will make for a good mark.

3. Narrowing down

The mark usually chooses themselves. The psychic makes a statement and points towards a row, quickly altering their gesture based on somebody responding visible to the statement. This makes it look like they pointed at the mark right from the beginning.

The mark is that way primed from the start to believe the psychic. They’re off-guard. Usually a bit surprised and totally unprepared for the quick burst of questions the psychic offers next. If those questions land and draw the mark in, they are followed by the actual reading. Otherwise, they move on and try again.

4. Testing the mark—Cold reading using subjective validation

The con—cold reading—hinges on a quirk of human psychology: if we personally relate to a statement, we will generally consider it to be accurate.

This unfortunate side effect of how our mind functions is called subjective validation.

Subjective validation, sometimes called personal validation effect, is a cognitive bias by which people will consider a statement or another piece of information to be correct if it has any personal meaning or significance to them. People whose opinion is affected by subjective validation will perceive two unrelated events (i.e., a coincidence) to be related because their personal beliefs demand that they be related.

As a consequence, many people will interpret even the most generic statement as being specifically about them if they can relate to what was said.

The more eager they are to find meaning in the statement, the stronger the effect.

The more they believe in the speaker’s ability to make accurate statements, the stronger the effect.

The basic mechanism of the psychic’s con is built on the mark being willing and able to relate what was said to themselves, even if it’s unintentional.

5. The subjective validation loop using validation statements

The psychic taps into this cognitive bias by making a series of statements that are tailored to be personally relatable—sound specific to you—while actually being statistically generic.

These statements come in many types. I use “validation statements” here as an umbrella term for all these various tactics.

Some common examples:

• Forer or Barnum statements are probably the most famous kind of statement that plays into the subjective validation effect. Many of these statements are inherently meaningless but are nonetheless felt to be accurate by listeners. Most people will consider “you tend to be hard on yourself” to be an accurate description of themselves, for example.
• Vanishing negative is where a question is rephrased to include a negative such as “not” or “don’t”. If the psychic asks “you don’t play the piano?” then they will be able to reframe the question as accurate after the fact, no matter what the answer is. If you answer negative: “didn’t think so”. Positive: “that’s what I thought.”
• Rainbow ruse where the psychic associates the mark with both a trait and its opposite. “You’re a very calm person, but if provoked you can get very angry.”
• Statistical guesses. Statements like “you have, or used to have, a scar on your left leg or knee” apply to almost everybody. With enough knowledge of common statistics, the psychic can make general statements that sound incredibly specific to the mark.
• Demographic guesses. Similar to statistical guesses, these are statements that are common to a demographic but will sound very specific to the mark that’s listening.
• Unverifiable predictions. Predictions like “somebody bears a strong ill will towards you but they are unlikely to act on it” are impossible to verify, but will sound true to many people.
• Shotgunning is one of the more common tactic where the psychic will fire off a series of statements. The mark will find one of the statements to be accurate and, due to how our minds work, will come away only remembering the correct statement.

An important part of this process is the tone and bearing of the psychic. They need to be confident, be quick in dismissing errors and moving on when they make mistakes, and they need to be quick to read people’s expressions and body language and adjust their responses to match.

6. The con is completed

At the end of the process, the mark is likely to remember that the reading was eerily correct—that the psychic had an almost supernatural accuracy—which primes them to become even more receptive the next time they attend.

This is where the con often becomes insidious: the effect becomes stronger the more cooperative the mark is, and they often become more cooperative over time.

What’s more, susceptibility has nothing to do with intelligence.

Somebody raised to believe they have high IQ is more likely to fall for this than somebody raised to think less of their own intellectual capabilities. Subjective validation is a quirk of the human mind. We all fall for it. But if you think you’re unlikely to be fooled, you will be tempted instead to apply your intelligence to “figure out” how it happened. This means you can end up using considerable creativity and intelligence to help the psychic fool you by coming up with rationalisations for their “ability”. And because you think you can’t be fooled, you also bring your intelligence to bear to defend the psychic’s claim of their powers. Smart people (or, those who think of themselves as smart) can become the biggest, most lucrative marks.

Whereas the sceptic who thinks less of themselves is more likely to just go:

“That’s a neat trick. I don’t know how you pulled it off. Must be very clever.”

And just move on.

Many psychics fool themselves

It isn’t unusual for psychics to unconsciously develop a practice of cold reading subconsciously. The psychics themselves might not even be aware of their own tactics.

As Denis Dutton describes:

As a postgraduate student in pursuit of a scientific career, he became intrigued with astrology. Though during this period he had nagging doubts about the physical basis of astrology, he was encouraged to continue with it by his many satisfied clients, who invariably found his readings “amazingly accurate” in describing their personal situations and problems. Not until he had one day obtained such a gratifying reaction to a horoscope which, he realized later, he had cast completely incorrectly, did he begin slowly to understand the real nature of his activity: his great success as an astrologer had nothing whatsoever to do with the validity of astrology as a science. He had become, in fact, a proficient cold reader, one who sincerely believed in the power of astrology under the constant reinforcement of his clients. He was fooling them, of course, but only after falling for the illusion himself.

There are many examples of this easily found once you start doing the research. The mechanism is simple enough and already baked into people’s preconceptions of how readings work so many psychics accidentally develop the knack for it, meaning that they’re not just conning the person being read, they are also conning themselves.

This point will become important later.

The LLMentalist Effect

1. The Audience Selects Itself
People sceptical about "AI" chatbots are less likely to use them. Those who actively don't disbelieve the possibility of chatbot "intelligence" won't get pulled in by the bot. The most active audience will be early adopters, tech enthusiasts, and genuine believers in AGI who will all generally be less critical and more open-minded.

2. The Scene is Set
Users are primed by the hype surrounding the technology. The chat environment sets the mood and expectations. Warnings about it being “early days” and “hallucinations” both anthropomorphise the bot and provide ready-made excuses for when one of its constant failures are noticed.

3. The Prompt Establishes the Context
Each user gives the chatbot a prompt and it answers. Many will either accept the answer as given or repeat variations on the initial prompt to get the desired result. They move on without falling for the effect. But some users engage in conversation and get drawn in.

4. The Marks Test Themselves
The chatbot’s answers sound extremely specific to the current context but are in fact statistically generic. The mathematical model behind the chatbot delivers a statistically plausible response to the question. The marks that find this convincing get pulled in.

5. The Subjective Validation Loop
The mark asks a series of questions and all of the replies sound like reasoned answers specific to the context but are in reality just statistically probable guesses. The more the mark engages, the more convinced they are of the chatbot’s intelligence.

6. “Wow! This chatbot thinks! It has sparks of general intelligence!”
The mark is left with the sense that the chatbot is uncannily close to being self-aware and that it is definitely capable of reasoning But it’s nothing more than a statistical and psychological effect.

1. The audience selects itself

If you aren’t interested in “AI”, you aren’t going to use an “AI” chatbot, and if you try one, you’re less likely to return.

This means that many of the avid users of these chatbots are self-selected to be enthusiastic and open-minded about the field of AI and the notion of Artificial General Intelligence (AGI)—that these technologies might lead to self-aware and self-improving reasoning systems.

Those who are genuine enthusiasts about AGI—that this field is about to invent a new kind of mind—are likely to be substantially more enthusiastic about using these chatbots than the rest.

This parallels the audience selection for the psychic’s con. Those who believe in an afterlife and that it can be contacted by the living are substantially more likely to attend a psychic’s reading than others.

2. Setting the stage

Our current environment of relentless hype sets the stage and builds up an expectation for at least glimmers of genuine intelligence. For all the warnings vendors make about these systems not being general intelligences, those statements are always followed by either an implied or an actual “yet”. The hype strongly implies that these are “almost” intelligences and that you should be able to perceive “sparks” of intelligence in them.

Those who believe are primed for subjective validation.

The warnings also play a role in setting the stage. “It’s early days” means that when the statistically generic nature of the response is spotted, it’s easily dismissed as an “error”. Anthropomorphising concepts such as using “hallucination” as a term help dismiss the fact that statistical responses are completely disconnected from meaning and facts. The hype and mythology of AI primes the audience to think of these systems as persons to be understood and engaged with, all but guaranteeing subjective validation.

3. The prompt establishes the context

The initial prompt interaction is the first filter. Most will just take the first answer and leave, or at most will repeat variations of their prompt until they get the result they wanted. These interactions are purely mechanical. The end-user is treating the chatbot merely as a generative widget, so they never get pulled into the LLMentalist effect.

Some of the end-users, usually those who are more enthusiastic about the prospect of “AI”, begin to engage and get pulled into “conversation” with a mathematical language model.

4. The mark tests themselves—subjective validation kicks in

That conversation is the primary filter. Those who want to believe will see the responses to their prompt as being both specifically about them and intelligent. They are primed to see the chatbot as a person that is reading their texts and thoughtfully responding to them. But that isn’t how language models work. LLMs model the distribution of words and phrases in a language as tokens. Their responses are nothing more than a statistically likely continuation of the prompt.

You give it text. It gives you a response that matches responses that texts like yours commonly get in its training data set.

Already, this is working along the same fundamental principle as the psychic’s con: the LLM isn’t “reading” your text any more than the psychic is reading your mind. They are giving you statistically plausible responses based on what you say. You’re the one finding ways to validate those responses as being specific to you as the subject of the conversation.

Because of how large the training data set is, the responses from the chatbot will look extremely convincing and specific, even though they are statistically generic. Once you’ve trained on most of the past twenty years of the web, large collections of stolen ebooks, all of Reddit, most of social media, and a substantial amount of custom interactions by low-wage workers, the model will have a response for almost everything you can think of, or can use a variation of something it’s already seen.

These initial interactions can be quite compelling, especially if you’re a believer in “AI”, but it is in the longer and repeated conversations that the effect really begins to kick in.

5. The subjective validation loop—RLHF enters the picture

It’s important to remember at this stage how Reinforcement Learning through Human Feedback works.

This is the method that vendors use to turn a raw language model into a chatbot that can hold a conversation.

RLHF doesn’t let the vendor make specific corrections to an LLM’s output. The method involves using human feedback to rank a variety of texts generated by the model, usually following some other form of fine-tuning. The ranked texts are in turn used to train a separate reward model. It’s this model that is responsible for the actual Reinforcement Learning of the LLM. The reward model, coupled with fine-tuning the LLM on collections of chats, is what turns the borderline unhinged conversations of a regular model into the fluent experience you see in systems such as ChatGPT.

Because the feedback is based on rankings, it can’t easily be based on specific issues. If a model makes a false statement in a conversation, that conversation gets a lower rank.

This lack of concrete specificity likely means that RLHF models in general are likely to reward responses that sound accurate. As the reward model is likely just another language model, it can’t reward based on facts or anything specific, so it can only reward output that has a tone, style, and structure that’s commonly associated with statements that have been rated as accurate.

Even the ratings themselves are suspect. Most, if not all, of the workers who provide this feedback to AI vendors are low-paid workers who are unlikely to have specialised knowledge relevant to the topic they’re rating, and even if they do, they are unlikely to have the time to fact-check everything.

That means they are going to be ranking the conversations almost entirely based on tone and sentence structure.

This is why I think that RLHF has effectively become a reward system that specifically optimises language models for generating validation statements: Forer statements, shotgunning, vanishing negatives, and statistical guesses.

In trying to make the LLM sound more human, more confident, and more engaging, but without being able to edit specific details in its output, AI researchers seem to have created a mechanical mentalist.

Instead of pretending to read minds through statistically plausible validation statements, it pretends to read and understand your text through statistically plausible validation statements.

The validation loop can continue for a while, with the mark constantly doing the work of convincing themselves of the language model’s intelligence. Done long enough, it becomes a form of reinforcement learning for the mark.

6. The marks become cheerleaders

The most enthusiastic believers in an imminent AI revolution are starting to sound very similar to long-time believers in psychics and mind-reading.

They come up with increasingly convoluted ideas and models to explain why the impossible is possible. They become more and more dismissive of fields of science and research that challenge their world view. Their own statements become tinged with awe and dread.

And they keep evangelising. This is real!

Often followed by: This is dangerous!

Remember, the effect becomes more powerful when the mark is both intelligent and wants to believe. Subjective validation is based on how our minds work, in general, and is unaffected by your reported IQ.

If anything, your intelligence will just improve your ability to rationalise your subjective validation and make the effect stronger. When it’s coupled with a genuine desire to believe in the con—that we are on the verge of discovering Artificial General Intelligence—the effect should both be irresistible and powerful once it takes hold.

This is why you can’t rely on user reports to discover these issues. People who believe in psychics will generally have only positive things to say about a psychic, even as they’re being bilked. People who believe we’re on the verge of building an AGI will only have positive things to say about chatbots that support that belief.

It’s easy to fall for this

Falling for this statistical illusion is easy. It has nothing to do with your intelligence or even your gullibility. It’s your brain working against you. Most of the time conversations are collaborative and personal, so your mind is optimised for finding meaning in what is said under those circumstances. If you also want to believe, whether it’s in psychics or in AGI, your mind will helpfully find reasons to believe in the conversation you’re having.

Once you’re so deep into it that you’ve done a press tour and committed yourself as a public figure to this idea, dislodging the belief that we now have a proto-AGI becomes impossible. Much like a scientist publicly stating that they believe in a particular psychic, their self-image becomes intertwined with their belief in that psychic. Any dismissal of the phenomenon will feel to them like a personal attack.

The psychic’s con is a mechanism that has been extraordinarily successful at fooling people over the years. It works.

The best defence is to respond the same way as you would to a convincing psychic’s reading: “That’s a neat trick, I wonder how they pulled it off?”

Well, now you know.

Once you’re aware of the fallibility of how your mind works, you should have an easier time spotting when that fallibility is being exploited, intentionally or not.

That brings us to an important question.

Is this intentional?

Given that there are billions of dollars at stake in the tech industry, it would be tempting to assume that the statistical illusion of intelligence was intentionally created by people in the tech industry.

I personally think that’s extraordinarily unlikely.

A popular response to various government conspiracy theories is that government institutions just aren’t that good at keeping secrets.

Well, the tech industry just isn’t that good at software. This illusion is, honestly, too clever to have been created intentionally by those making it.

The field of AI research has a reputation for disregarding the value of other fields, so I’m certain that this reimplementation of a psychic’s con is entirely accidental. It’s likely that, being unaware of much of the research in psychology on cognitive biases or how a psychic’s con works, they stumbled into a mechanism and made chatbots that fooled many of the chatbot makers themselves.

Remember what I wrote above about psychics frequently having conned themselves, that many of them aren’t even aware of their own scam?

The same applies here. I think this is an industry that didn’t understand what it was doing and, now, doesn’t understand what it did.

That’s why so many people in tech are completely and utterly convinced that they have created the first spark of true Artificial General Intelligence.

This new era of tech seems to be built on superstition and pseudoscience

Once I started to research the possibility that LLM interactions were a variation on the psychic’s con, I began to see parallels everywhere in the field of “AI”.

• Hooking a language model up to an MRI and claiming that it can read minds.
• Claiming to be able to discern criminality based on facial expressions and gait.
• Proposing magical solutions to health problems.
• Literal predictions of the future.
• Claiming to be able to discern the honesty of potential employees.

All of these are proposed applications of “AI” systems, but they are also all common psychic scams. Mind reading, police assistance, faith healing, prophecy, and even psychic employee vetting are all right out of the mentalist playbook.

Even though I have no doubts that these efforts are sincere, it’s becoming more and more obvious that the tech industry has given itself wholesale to superstition and pseudoscience. They keep ignoring the warnings coming from other fields and the concerns from critics in their own camp.

Large Language Models don’t have the functionality or features to make up for this wave of superstition.

• “Hallucinations” are a pervasive flaw that’s baked into how LLMs work.
• Summarisations are error-prone and prone to generalising about the text being summarised.
• Their “reasoning” is a statistical illusion.
• Their performance at natural language processing tasks is only marginally better than that of smaller language models.
• They tend to memorise and copy text without attribution.

Taken together, these flaws make LLMs look less like an information technology and more like a modern mechanisation of the psychic hotline.

Delegating your decision-making, ranking, assessment, strategising, analysis, or any other form of reasoning to a chatbot becomes the functional equivalent to phoning a psychic for advice.

Imagine Google or a major tech company trying to fix their search engine by adding a psychic hotline to their front page? That’s what they’re doing with Bard.

—“Our university students can’t make heads nor tails of our website. Let’s add a psychic hotline!”

—“We need to improve our customer service portal. Let’s add a psychic hotline!”

—“We’ve added a psychic hotline button to your web browser! No, you can’t get rid of it. You’re welcome!”

—“Can’t understand a thing in our technical docs? Refer to our fancy new psychic hotline!”

The AI bubble is going to be a tough one to weather.

More on “AI”

I’ve spent some time writing about the many flaws of language models and generative “AI”.

• I’ve written about how language models are a backward-facing tool in a novelty-seeking industry and why I think using language models for programming is a bad idea.
• “AI” summaries are inherently unreliable.
• Their tendency towards shortcuts makes them dangerous in healthcare.
• Most of the research indicating a productivity benefit to “AI” is, at best, flawed, and at worst are completely detached from the reality of modern office work.
• AI vendors have a history of pseudoscience and snake oil.
• Even if you do think that a language model’s unsolvable tendency towards ‘hallucinations’ doesn’t disqualify the technology from replacing search engines, the many security issues that language models suffer from should. The “write a prompt; get the output” model is inherently insecure. These systems are also vulnerable to a form of keyword manipulation exploit that’s impossible to prevent.

I’ve come to the conclusion that a language model is almost always the wrong tool for the job.

I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.

If you do have to use generative AI, either because it’s a mandate from above your pay grade or some other requirement, I have written a book that’s specifically about the issues with using generative “AI” for work:

The Intelligence Illusion: a practical guide to the business risks of Generative AI.

It’s only $35 USD for EPUB and PDF, which is only 15% of the $240 USD cost of twelve months of ChatGPT Plus.

But, again, I’d much rather you just avoid using a language model in the first place and save both the cost of the ebook and the ChatGPT subscription.

References on the Psychic’s Con

• Cold reading (Wikipedia)
• How to Become Psychic and Cold Read People
• A Simple Introduction to Cold Reading
• Cold reading (Rational Wiki)
• 7 Tricks Psychics Bullshit People With That Everyone Should Know
• Should You Believe in Psychics? Psychology and logic join forces to debunk psychics (Psychology Today)
• Motivated reasoning (Wikipedia)
• Cold Reading: How I Made Others Believe I Had Psychic Powers
• Cold reading (Sceptic’s Dictionary)
• Subjective validation (Sceptic’s Dictionary)
• Subjective validation (Wikipedia)
• Coincidences: Remarkable or Random?
• Psychic Experiences: Psychic Illusions
• Guide to Cold Reading
• The Cold Reading Technique
• Forer effect(Sceptic’s Dictionary)
• Tricks of the Psychic Trade (Psychology Today)
• Psychic Scams
• Ten Tricks of the Psychics I Bet You Didn’t Know (You Won’t Believe #6!)

This is why the first draft is often the hardest and why everybody dreads it. That’s the part where you dive into yourself and dredge up the truth itself.

Of course everybody wants to skip that. They want the residue that writing leaves behind—text—because that petrification of metaphors^[1] is valued as a proxy for thought. That value is both cultural and economic, so people want it, seek it out, and pay for it. Writing done well will lead others to follow your own path of thinking. Reading becomes thinking as well in process that is a mirror image to that of writing. The text detritus that the writing process leaves behind helps sell what needs to be sold, market what needs to be marketed, and guide what needs to be guided.

But, in a slight twist on the usual aphorism, it’s only the thought that counts. Everything else gets value from proximity.

This process is inherently unnerving when it works because the clarity it affords can be painful.

That clear sight fades quickly as rationalisation sets in, but if you get into the habit of writing you can keep the ghosts of self-construction at bay—at least at enough of a distance to afford some space to negotiate with them.

This is what everybody is so eager to lose, because all that matters to them is the economic value of the textual byproduct.

Fair, given that economic value is what pays the bills and keeps you housed and fed. But that desire for efficiency and productivity risks mistaking the output for the process.

The value in writing lies in what we discover while writing.

Auto-generating text based on other people’s discoveries and then automatically summarising that text by finding commonalities with existing text creates a loop of mechanised nonsense.

It’s a prayer wheel for capitalism.

A mechanical recitation of static money mantras that paid off it the past. Spin the wheel and the money will flow again because obviously an inner life can’t have any value.

Obviously an inner life has no value.

This isn’t the newsletter I had planned.

Even though writing prose is, directly and indirectly, a major source of income for me, readers aren’t why I write.

I’ve been working on a few essays that require both focus and clarity. Analyses and critiques. All stuff that matters. Just stuff that doesn’t matter to me today. They’re thoughts that require exertion and effort, but contain little of today’s truth. Maybe they’ll congeal into cohesive thoughts next week. Maybe they won’t.

This newsletter was supposed to be different, but that wasn’t where my mind is at.

My thoughts rarely feel clear. This is why I write. Unblanking the page is a process that, for a few moments, draws the curtains back and lets the sun in.

That light reveals, and today my thoughts are on those who seem eager to bargain away that revelation.

I can’t help but feel that it will make the world a darker place.

Now, most of the questions are standard ones—lines of inquiry you might find recommended in books such as How to Read a Book—but there are a couple of additional questions that become especially useful in a world dominated by social media:

1. What does the author want to believe?
2. What’s the game?

What does the author want to believe?

The first question is the more complex one. What the author wants the reader to believe is usually straightforward. It’s what they’re trying to argue in what you’re reading. But, what the author wants to believe themselves is often the more important one in an age of misinformation.

For example, somebody who works for Facebook is going to want to believe a few things:

• That it’s possible to work for Facebook and be a good person.
• That the work and research they’re doing is worthwhile.
• That the decisions they’ve made at work are the correct ones.

These are all beliefs that are necessary for them to protect their emotional wellbeing, which means that everything they write is going to add to the ramparts of their psychological self-defence. Very few non-authors are going to intentionally make themselves more vulnerable and fragile with their writing.

The bias this creates is a little bit different from the unconscious biases we all have in that these biases are a pre-emptive defence mechanism and people can get quite aggressive if they’re challenged. This can also lead people to rationalise themselves into strange intellectual corners.

1. “Crypto-coins are an interesting technology that could be used for social good!”
2. “Oh, no. There’s a lot of fraud going on.”
3. “But I’m a good person who doesn’t support fraud, so there must be something else going on.”
4. “Well, of course, if the finance industry and regulators are biased against crypto, then they’re going to drive legitimate users away and all you’re left with is fraud.”
5. “Crypto is being destroyed by a conspiracy of global elites!”

Bonkers opinions, arrived at step-by-step out of an intense drive to protect their conception of themselves.

You can see this playing out in AI already. It starts with “AI is going to lead to AGI and that’s going to lead to a lot of good!”, and where it ends is with bonkers opinions, spouted in public.

What’s the game?

The other question is to ask yourself what the rules are for the game that the writer is playing.

There’s always a game. Some of them are constructive.

You have people, such as web developers, writing about their field in order to connect with others in their field. Writing or creating media usually isn’t a part of their job description. The game is a mechanism for being a part of something bigger. The rewards in the game are social, not monetary.

Tangential to this are the people who play the social media drama game. You always have a few types who are doing it for the attention and the kind of attention doesn’t really matter that much, just the volume.

Some of them are less constructive, such as the grifters who don’t care about the field but are just in it to earn money. These are the types that will go from gig economy hustle, to crypto, to AI in a heartbeat. The actual activity is irrelevant to their game. What matters to them are the rewards.

Then you have the types like me, who are towing the fine line of making media that’s hopefully useful to a field, but without separating yourself from the field proper like the hustlers and grifters do. Educators, trainers, journalists, and writers all contribute to a given field by doing the practical research and documentation work that is very, very hard to pull off as a hobby or sideline. We try to employ sales tactics that are proven to work while avoiding the toxic methods employed by the grifters.

It can get tricky, but we’re helped by the fact that grifters are usually over-the-top enthusiastic about the grift they’re pulling, and that they tend to be a bit disconnected from the work being done in the fields themselves.

These two questions are a useful rubric that helps you assess much of what you read.

It helps you contextualise the writing and understand where it’s coming from.

More importantly, it helps you assess distance. As in, “how distant is this from my own world view and practice?”

Because text that’s distant, that might as well belong to another world, is often neither wrong nor right for your context.

And “not even wrong” is usually something you can just ignore.

What’s my game?

A few weeks ago I was interviewed for an article. Most of it was to give the journalist some technical context for much of today’s AI discourse. I like to think I did a decent job of providing that context, but I refer you to what I wrote above about what the author would like to believe.

I ended up getting quoted in the final piece, much to my parents’ joy, but I found the experience overall a bit disconcerting. The interview itself gave me odd vibes at the time.

The piece itself is well-made and does a very good job, considering how firmly mainstream US media has bought into the AI bubble.

It threads the needle between letting the AI industry’s executives say what they want to say without overtly implying that they have an ulterior motive, while still framing what they say as something that probably has ulterior motives.

But…

Given the demographics of those who have done the most work in highlighting the many issues with language and diffusion models, I have to wonder what sort criteria or biases led to me—a cis, white, middle-aged dude—being cited as a critic of the technology over others who have done much, much more work on it than I have.

The obvious answer is an uncomfortable one, but one I find impossible to dismiss.

It leaves me questioning what I’m doing when it comes to covering AI and machine learning.

I have no interest in becoming a professional “AI critic”. Others do that job much better than I can. I did my research on generative AI because I work in software development and wanted to get a clear idea of its impact on businesses and software development. That research led to the book and to last week’s essay on the impact I think language models will have on software development.

My beat is software development—coding and managing—not AI. It just so happens that language models are seen as the next big thing that will transform my industry, so I have to cover it if I am to do my job.

I think that the only way to square this circle is to take a firm line on not participating in AI discourse in general media. I need the freedom to be a bit more casual—a bit exploratory—in my own spaces such as my newsletter or blog, but elsewhere the line has to be that my beat is software development and how it’s affected by language models, not AI in general.

Of course, I’d make an exception for local Icelandic media, but that’s mostly out of self-preservation. I don’t think it would do us here in Iceland any good to fall for yet another bubble.

But, other than that, I need to be clear in my own mind what game I’m playing: I’m a software developer researching and writing about issues with coding and managing software development projects. Currently, many of those issues have to do with the AI bubble, but not all of them.

That’s me. That’s my game.

I’m not a pundit, full-time AI critic, AI researcher, or activist. That’s not my game.

It might become a fine line to toe, at times, as the AI bubble escalates, but I’m hoping that being open and transparent about it—about my worries and concerns—will make the task easier in the long run.

And, if I change my mind, I’ll let you know.

Many thanks to all of those who attended. The references for the presentation are also the references for this essay, which you can find all the way down in the footnotes section.

The best way to support this newsletter or my blog is to buy one of my books, The Intelligence Illusion: a practical guide to the business risks of Generative AI or Out of the Software Crisis. Or, you can buy them both as a bundle.

The software industry is very bad at software

Here’s a true story. Names withheld to protect the innocent.

A chain of stores here in Iceland recently upgraded their point-of-sale terminals to use new software.

Disaster, obviously, ensued. The barcode scanner stopped working properly, leading customer to be either overcharged or undercharged. Everything was extremely slow. The terminals started to lock up regularly. The new invoice printer sucked. A process that had been working smoothly was now harder and took more time.

The store, where my “informant” is a manager, deals with a lot of businesses, many of them stores. When they explain to their customers why everything is taking so long, their answer is generally the same:

“Ah, software upgrade. The same happened to us when we upgraded our terminals.”

This is the norm.

The new software is worse in every way than what it’s replacing. Despite having a more cluttered UI, it seems to have omitted a bunch of important features. Despite being new and “optimised”, it’s considerably slower than what it’s replacing.

This is also the norm.

Switching costs are, more often than not, massive for business software, and purchases are not decided by anybody who actually uses it. The quality of the software disconnects from sales performance very quickly in a growing software company. The company ends up “owning” the customer and no longer has any incentive to improve the software. In fact, because adding features is a key marketing and sales tactic, the software development cycle becomes an act of intentional, controlled deterioration.

Enormous engineering resources go into finding new ways to minimise the deterioration—witness Microsoft’s “ribbon menu”, a widget invented entirely to manage the feature escalation mandated by marketing.

This is the norm.

This has always been the norm, from the early days of software.

The software industry is bad at software. Great at shipping features and selling software. Bad at the software itself.

Why I started researching “AI” for programming

In most sectors of the software industry, sales performance and product quality are disconnected.

By its nature software has enormous margins which further cushion it from the effect of delivering bad products.

The objective impact of poor software quality on the bottom lines of companies like Microsoft, Google, Apple, Facebook, or the retail side of Amazon is a rounding error. The rest only need to deliver usable early versions, but once you have an established customer base and an experienced sales team, you can coast for a long, long time without improving your product in any meaningful way.

You only need to show change. Improvements don’t sell, it’s freshness that moves product. It’s like store tomatoes. Needs to look good and be fresh. They’re only going to taste it after they’ve paid, so who cares about the actual quality.

Uptime reliability is the only quality measurement with a real impact on ad revenue or the success of enterprise contracts, so that’s the only quality measurement that ultimately matters to them.

Bugs, shoddy UX, poor accessibility—even when accessibility is required by law—are non-factors in modern software management, especially at larger software companies.

The rest of us in the industry then copy their practices, and we mostly get away with it. Our margins may not be as enormous as Google’s, but they are still quite good compared to non-software industries.

We have an industry that’s largely disconnected from the consequences of making bad products, which means that we have a lot of successful but bad products.

The software crisis

Research bears this out. I pointed out in my 2021 essay Software Crisis 2.0 that very few non-trivial software projects are successful, even when your benchmarks are fundamentally conservative and short term.

For example, the following table is from a 2015 report by the Standish Group on their long term study in software project success:

This is based on data that’s collected and anonymised from a number of organisations in a variety of industries. You’ll note that very few projects outright succeed. Most of them go over budget or don’t deliver the functionality they were supposed to. A frightening number of large projects outright fail to ship anything usable.

In my book Out of the Software Crisis, I expanded on this by pointing out that there are many classes and types of bugs and defects that we don’t measure at all, many of them catastrophic, which means that these estimates are conservative. Software project failure is substantially higher than commonly estimated, and success if much rarer than the numbers would indicate.

The true percentage of large software projects that are genuinely successful in the long term—that don’t have any catastrophic bugs, don’t suffer from UX deterioration, don’t end up having core issues that degrade their business value—is probably closer to 1–3%.

The management crisis

We also have a management crisis.

The methods of top-down-control taught to managers are counterproductive for software development.

• Managers think design is about decoration when it’s the key to making software that generates value.
• Trying to prevent projects that are likely to fail is harmful for your career, even if the potential failure is wide-ranging and potentially catastrophic.
• When projects fail, it’s the critics who tried to prevent disaster who are blamed, not the people who ran it into the ground.
• Supporting a project that is guaranteed to fail is likely to benefit your career, establish you as a “team player”, and protects you from harmful consequences when the project crashes.
• Teams and staff management in the software industry commonly ignores every innovation and discovery in organisational psychology, management, and systems-thinking since the early sixties and operate mostly on management ideas that Henry Ford considered outdated in the 1920s.

We are a mismanaged industry that habitually fails to deliver usable software that actually solves the problems it’s supposed to.

Thus, Weinberg’s Law:

If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

It’s into this environment that “AI” software development tools appear.

The punditry presented it as a revolutionary improvement in how we make software. It’s supposed to fix everything.

—This time the silver bullet will work!

Because, of course, we have had such a great track record with silver bullets.

So, I had to dive into it, research it, and figure out how it really worked. I needed to understand how generative AI works, as a system. I haven’t researched any single topic to this degree since I finished my PhD in 2006.

This research led me to write my book The Intelligence Illusion: a practical guide to the business risks of Generative AI. In it, I take a broader view and go over the risks I discovered that come with business use of generative AI.

But, ultimately, all that work was to answer the one question that I was ultimately interested in:

Is generative AI good or bad for software development?

To even have a hope of answering this, we first need to define our terms, because the conclusion is likely to vary a lot depending on how you define “AI” or even "software development.

A theory of software development as an inclusive system

Software development is the entire system of creating, delivering, and using a software project, from idea to end-user.

That includes the entire process on the development side—the idea, planning, management, design, collaboration, programming, testing, prototyping—as well as the value created by the system when it has been shipped and is being used.

My model is that of theory-building. From my essay on theory-building, which itself is an excerpt from Out of the Software Crisis:

Beyond that, software is a theory. It’s a theory about a particular solution to a problem. Like the proverbial garden, it is composed of a microscopic ecosystem of artefacts, each of whom has to be treated like a living thing. The gardener develops a sense of how the parts connect and affect each other, what makes them thrive, what kills them off, and how you prompt them to grow. The software project and its programmers are an indivisible and organic entity that our industry treats like a toy model made of easily replaceable lego blocks. They believe a software project and its developers can be broken apart and reassembled without dying.

What keeps the software alive are the programmers who have an accurate mental model (theory) of how it is built and works. That mental model can only be learned by having worked on the project while it grew or by working alongside somebody who did, who can help you absorb the theory. Replace enough of the programmers, and their mental models become disconnected from the reality of the code, and the code dies. That dead code can only be replaced by new code that has been ‘grown’ by the current programmers.

Design and user research is an integral part of the mental model the programmer needs to build, because none of the software components ultimately make sense without the end-user.

But, design is also vital because it is, to reuse Donald G. Reinertsen’s definition from Managing the Design Factory (p. 11), design is economically useful information that generally only becomes useful information through validation of some sort. Otherwise it’s just a guess.

The economic part usually comes from the end-user in some way.

This systemic view is inclusive by design as you can’t accurately measure the productivity or quality of a software project unless you look at it end to end, from idea to end-user.

• If it doesn’t work for the end-user, then it’s a failure.
• If the management is dysfunctional, then the entire system is dysfunctional.
• If you keep starting projects based on unworkable ideas, then your programmer productivity doesn’t matter.

Lines of code isn’t software development. Working software, productively used, understood by the developers, is software development.

A high-level crash course in language models

Language models, small or large, are today either used as autocomplete copilots or as chatbots. Some of these language model tools would be used by the developer, some by the manager or other staff.

I’m treating generative media and image models as a separate topic, even when they’re used by people in the software industry to generate icons, graphics, or even UIs. They matter as well, but don’t have the same direct impact on software quality.

To understand the role these systems could play in software development, we need a little bit more detail on what language models are, how they are made, and how they work.

Most modern machine learning models are layered networks of parameters, each representing its connection to its neighbouring parameters. In a modern transformer-based language model most of these parameters are floating point numbers—weights—that describe the connection. Positive numbers are an excitatory connection. Negative numbers are inhibitory.

These models are built by feeding data through a tokeniser that breaks text into tokens—often one word per token—that are ultimately fed into an algorithm. That algorithm constructs the network, node by node, layer by layer, based on the relationships it calculates between the tokens/words. This is done in several runs and, usually, the developer of the model will evaluate after each run that the model is progressing in the right direction, with some doing more thorough evaluation at specific checkpoints.

The network is, in a very fundamental way, a mathematical derivation of the language in the data.

A language model is constructed from the data. The transformer code regulates and guides the process, but the distributions within the data set are what defines the network.

This process takes time—both collecting and managing the data set and the build process itself—which inevitably introduces a cut-off point for the data set. For OpenAI and Anthropic, that cut-off point is in 2021. For Google’s PaLM2 it’s early 2023.

Aside: not a brain

This is very, very different from how a biological neural network interacts with data. A biological brain is modified by input and data—its environment—but its construction is derived from nutrition, its chemical environment, and genetics.

The data set, conversely, is a deep and fundamental part of the language model. The algorithm’s code provides the process while the weights themselves are derived from the data, and the model itself is dead and static during input and output.

The construction process of a neural network is called “training”, which is yet another incredibly inaccurate term used by the industry.

• A pregnant mother isn’t “training” the fetus.
• A language model isn’t “trained” from the data, but constructed.

This is nonsense.

But this is the term that the AI industry uses, so we’re stuck with it.

A language model is a mathematical model built as a derivation of its training data. There is no actual training, only construction.

This is also why it’s inaccurate to say that these systems are inspired by their training data. Even though genes and nutrition make an artist’s mind they are not in what any reasonable person would call “their inspiration”. Even when they are sought out for study and genuine inspiration, it’s our representations of our understanding of the genes that are the true source of inspiration. Nobody sticks their hand in a gelatinous puddle of DNA and spontaneously gets inspired by the data it encodes.

Training data are construction materials for a language models. A language model can never be inspired. It is itself a cultural artefact derived from other cultural artefacts.

The machine learning process is loosely based on decades-old grossly simplified models of how brains work.

A biological neuron is a complex system in its own right—one of the more complex cells in an animal’s body. In a living brain, a biological neuron will use electricity, multiple different classes of neurotransmitters, and timing to accomplish its function in ways that we still don’t fully understand. It even has its own built-in engine for chemical energy.

The brain as a whole is composed of not just a massive neural network, but also layers of hormonal chemical networks that dynamically modify its function, both granularly and as a whole.

The digital neuron—a single signed floating point number—is to a biological neuron what a flat-head screwdriver is to a Tesla.

They both contain metal and that’s about the extent of their similarity.

The human brain contains roughly 100 billion neuron cells, a layered chemical network, and a cerebrovascular system that all integrate as a whole to create a functioning, self-aware system capable of general reasoning and autonomous behaviour. This system is multiple orders of magnitude more complex than even the largest language model to date, both in terms of individual neuron structure, and taken as a whole.

It’s important to remember this so that we don’t fall for marketing claims that constantly imply that these tools are fully functioning assistants.

The prompt

After all of this, we have a data set which can be used to generate text in response to prompts.

Prompts such as:

Who was the first man on the moon?

The input phrase, or prompt, has no structure beyond the linguistic. It’s just a blob of text. You can’t give the model commands or parameters separately from other input. Because of this, if your model lets a third party enter text, an attacker will always be able to bypass whatever restrictions you put on it. Control prompts or prefixes will be discovered and countermanded. Delimiters don’t work. Fine-tuning the model only limits the harm, but doesn’t prevent it.

This is called a prompt injection and what it means is that model input can’t be secured. You have to assume that anybody that can send text to the model has full access to it.

Language models need to be treated like an unsecured client and only very carefully integrated into other systems.

The response

What you’re likely to get back from that prompt would be something like:

On July 20, 1969, Neil Armstrong became the first human to step on the moon.

This is NASA’s own phrasing. Most answers on the web are likely to be variations on this, so the answer from a language model is likely to be so too.

• The moon landing happens to be a fact, but the language model only knows it as a text.

The prompt we provided is strongly associated in the training data set with other sentences that are all variations of NASA’s phrasing of the answer. The model won’t answer with just “Neil Armstrong” because it isn’t actually answering the question, it’s responding with the text that correlates with the question. It doesn’t “know” anything.

• The language model is fabricating a mathematically plausible response, based on word distributions in the training data.
• There are no facts in a language model or its output. Only memorised text.

It only fabricates. It’s all “hallucinations” all the way down.

Occasionally those fabrications correlate with facts, but that is a mathematical quirk resulting from the fact that, on average, what people write roughly correlates with their understanding of a factual reality, which in turn roughly correlates with a factual reality.

A knowledge system?

To be able to answer that question and pass as a knowledge system, the model needs to memorise the answer, or at least parts of the phrase.

Because “AI” vendors are performing a sleight-of-hand here and presenting statistical language synthesis engines as knowledge retrieval systems, their focus in training and testing is on “facts” and minimising “falsehoods”. The model has no notion of either, as it’s entirely a language model, so the only way to square this circle is for the model to memorise it all.

• To be able to answer a question factually, not “hallucinate”, and pass as a knowledge system, the model needs to memorise the answer.
• The model doesn’t know facts, only text.
• If you want a fact from it, the model will need to memorise text that correlates with that fact.

“Dr. AI”?

Vendors then compound this by using human exams as benchmarks for reasoning performance. The problem is that bar exams, medical exams, and diagnosis tests are specifically designed to mostly test rote memorisation. That’s what they’re for.

The human brain is bad at rote memorisation and generally it only happens with intensive work and practice. If you want to design a test that’s specifically intended to verify that somebody has spent a large amount of time studying a subject, you test for rote memorisation.

Many other benchmarks they use, such as those related to programming languages also require memorisation, otherwise the systems would just constantly make up APIs.

• Vendors use human exams as benchmarks.
• These are specifically designed to test rote memorisation, because that’s hard for humans.
• Programming benchmarks also require memorisation. Otherwise, you’d only get pseudocode.

Between the tailoring of these systems for knowledge retrieval, and the use of rote memorisation exams and code generation as benchmarks, the tech industry has created systems where memorisation is a core part of how they function. In all research to date, memorisation has been key to language model performance in a range of benchmarks.^[1]

If you’re familiar with storytelling devices, this here would be a Chekhov’s gun. Observe! The gun is above the mantelpiece:

👉🏻👉🏻 memorisation!

Make a note of it, because those finger guns are going to be fired later.

Biases

Beyond question and answer, these systems are great at generating the averagely plausible text for a given prompt. In prose, current system output smells vaguely of sweaty-but-quiet LinkedIn desperation and over-enthusiastic social media. The general style will vary, but it’s always going to be the most plausible style and response based on the training data.

One consequence of how these systems are made is that they are constantly backwards-facing. Where brains are focused on the present, often to their detriment, “AI” models are built using historical data.

The training data encompasses thousands of diverse voices, styles, structures, and tones, but some word distributions will be more common in the set than others and those will end up dominating the output. As a result, language models tend to lean towards the “racist grandpa who has learned to speak fluent LinkedIn” end of the spectrum.^[2]

This has implications for a whole host of use cases:

• Generated text is going to skew conservative in content and marketing copy in structure and vocabulary. (Bigoted, prejudiced, but polite and inoffensively phrased.)
• Even when the cut-off date for the data set is recent, it’s still going to skew historical because what’s new is also comparatively smaller than the old.
• Language models will always skew towards the more common, middling, mediocre, and predictable.
• Because most of these models are trained on the web, much of which is unhinged, violent, pornographic, and abusive, some of that language will be represented in the output.

Modify, summarise, and “reason”

The superpower that these systems provide is conversion or modification. They can, generally, take text and convert it to another style or structure. Take this note and turn it into a formal prose, and it will! That’s amazing. I don’t think that’s a trillion-dollar industry, but it’s a neat feature that will definitely be useful.

They can summarise text too, but that’s much less reliable than you’d expect. It unsurprisingly works best with text that already provides its own summary, such as a newspaper article (first paragraphs always summarise the story), academic paper (the abstract), or corporate writing (executive summary). Anything that’s a mix of styles, voices, or has an unusual structure won’t work as well.

What little reasoning they do is entirely based on finding through correlation and re-enacting prior textual descriptions of reasoning. They fail utterly when confronted with adversarial or novel examples. They also fail if you rephrase the question so that it no longer correlates with the phrasing in the data set.^[3]

So, not actual reasoning. “Reasoning”, if you will. In other “AI” model genres these correlations are often called “shortcuts”, which feels apt.

To summarise:

• Language models are a mathematical expression of the training data set.
• Have very little in common with human brains.
• Rely on inputs that can’t be secured.
• Lie. Everything they output is a fabrication.
• Memorise heavily.
• Great for modifying text. No sarcasm. Genuinely good at this.
• Occasionally useful for summarisation if you don’t mind being lied to regularly.
• Don’t actually reason.

Why I believe “AI” for programming is a bad idea

If you recall from the start of this essay, I began my research into machine learning and language models because I was curious to see if they could help fix or improve the mess that is modern software development.

There was reason to be hopeful. Programming languages are more uniform and structured than prose, so it’s not too unreasonable to expect that they might lend themselves to language models. Programming language output can often be tested directly, which might help with the evaluation of each training run.

Training a language model on code also seems to benefit the model. Models that include substantial code in their data set tend to be better at correlative “reasoning” (to a point, still not actual reasoning), which makes sense since code is all about representing structured logic in text.

But, there is an inherent Catch 22 to any attempt at fixing software industry dysfunction with more software. The structure of the industry depends entirely on variables that everybody pretends are proxies for end user value, but generally aren’t. This will always tend to sabotage our efforts at industrial self-improvement.

The more I studied language models as a technology the more flaws I found until it became clear to me that odds are that the overall effect on software development will be harmful. The problem starts with the models themselves.

1. Language models can’t be secured

This first issue has less to do with the use of language models for software development and more to do with their use in software products, which is likely to be a priority for many software companies over the next few years.

Prompt injections are not a solved problem. OpenAI has come up with a few “solutions” in the past, but none of them actually worked. Everybody expects this to be fixed, but nobody has a clue how.

Language models are fundamentally based on the idea that you give it text as input and get text as output. It’s entirely possible that the only way to completely fix this is to invent a completely new kind of language model and spend a few years training it from scratch.

• A language model needs to be treated like an unsecured client. It’s about as secure as a web page form. It’s vulnerable to a new generation of injection vulnerabilities, both direct and indirect, that we still don’t quite understand.^[4]

The training data set itself is also a security hazard. I’ve gone into this in more detail elsewhere^[5], but the short version is that training data set is vulnerable to keyword manipulation, both in terms of altering sentiment and censorship.

Again, fully defending against this kind of attack would seem to require inventing a completely new kind of language model.

Neither of these issues affect the use of language models for software development, but it does affect our work because we’re the ones who will be expected to integrate these systems into existing websites and products.

2. It encourages the worst of our management and development practices

A language model will never question, push back, doubt, hesitate, or waver.

Your managers are going to use it to flesh out and describe unworkable ideas, and it won’t complain. The resulting spec won’t have any bearing with reality.

People on your team will do “user research” by asking a language model, which it will do even though the resulting research will be fiction and entirely useless.

It’ll let you implement the worst ideas ever in your code without protest. Ask a copilot “how can I roll my own cryptography?” and it’ll regurgitate a half-baked expression of sha1 in PHP for you.

Think of all the times you’ve had an idea for an approach, looked up how to do it on the web, and found out that, no, this was a really bad idea? I have a couple of those every week when I’m in the middle of a project.

Language models don’t deliver productivity improvements. They increase the volume, unchecked by reason.

A core aspect of the theory-building model of software development is code that developers don’t understand is a liability. It means your mental model of the software is inaccurate which will lead you to create bugs as you modify it or add other components that interact with pieces you don’t understand.

Language model tools for software development are specifically designed to create large volumes of code that the programmer doesn’t understand. They are liability engines for all but the most experienced developer. You can’t solve this problem by having the “AI” understand the codebase and how its various components interact with each other because a language model isn’t a mind. It can’t have a mental model of anything. It only works through correlation.

These tools will indeed make you go faster, but it’s going to be accelerating in the wrong direction. That is objectively worse than just standing still.

3. Its User Interfaces do not work, and we haven’t found interfaces that do work

Human factors studies, the field responsible for designing cockpits and the like, discovered that humans suffer from an automation bias.

What it means is that when you have cognitive automation—something that helps you think less—you inevitably think less. That means that you are less critical of the output than if you were doing it yourself. That’s potentially catastrophic when the output is code, especially since the quality of the generated code is, understandably considering how the system works, broadly on the level of a novice developer.^[6]

Copilots and chatbots—exacerbated by anthropomorphism—seem to trigger our automation biases.

Microsoft themselves have said that 40% of GitHub Copilot’s output is committed unchanged.^[7]

Let’s not get into the question of how we, as an industry, put ourselves in the position where Microsoft can follow a line of code from their language model, through your text editor, and into your supposedly decentralised version control system.

People overwhelmingly seem to trust the output of a language model.

If it runs without errors, it must be fine.

But that’s never the case. We all know this. We’ve all seen running code turn out to be buggy as hell. But something in our mind switches off when we use tools for cognitive automation.

4. It’s biased towards the stale and popular

The biases inherent in these language models are bad enough when it comes to prose, but they become a functional problem in code.

• Its JS code will lean towards React and node, most of it several versions old, and away from the less popular corners of the JS ecosystem.
• The code is, inevitably, more likely to be built around CommonJS modules instead of the modern ESM modules.
• It won’t know much about Deno or Cloudflare Workers.
• It’ll always prefer older APIs over new. Most of these models won’t know about any API or module released after 2021. This is going to be an issue for languages such as Swift.
• New platforms and languages don’t exist to it.
• Existing data will outweigh deprecations and security issues.
• Popular but obsolete or outdated open source projects will always win out over the up-to-date equivalent.

These systems live in the popular past, like the middle-aged man who doesn’t realise he isn’t the popular kid at school any more. Everything he thinks is cool is actually very much not cool. More the other thing.

This is an issue for software because our industry is entirely structured around constant change. Software security hinges on it. All of our practices are based on constant march towards the new and fancy. We go from framework to framework to try and find the magic solution that will solve everything. In some cases language models might help push back against that, but it’ll also push back against all the very many changes that are necessary because the old stuff turned out to be broken.

• The software industry is built on change.
• Language models are built on a static past.

5. No matter how the lawsuits go, this threatens the existence of free and open source software

Many AI vendors are mired in lawsuits.^[8]

These lawsuits all concentrate on the relationship between the training data set and the model and they do so from a variety of angles. Some are based on contract and licensing law. Others are claiming that the models violate fair use. It’s hard to predict how they will go. They might not all go the same way, as laws will vary across industries and jurisdictions.

No matter the result, we’re likely to be facing a major decline in the free and open source ecosystem.

1. All of these models are trained on open source code without payment or even acknowledgement, which is a major disincentive for contributors and maintainers. That large corporations might benefit from your code is a fixture of open source, but they do occasionally give back to the community.
2. Language models—built on open source code—commonly replace that code. Instead of importing a module to do a thing, you prompt your Copilot. The code generated is almost certainly based on the open source module, at least partially, but it has been laundered through the language model, disconnecting the programmer from the community, recognition, and what little reward there was.

Language models demotivate maintainers and drain away both resources and users. What you’re likely to be left with are those who are building core infrastructure or end-user software out of principle. The “free software” side of the community is more likely to survive than the rest. The Linux kernel, Gnome, KDE—that sort of thing.

The “open source” ecosystem, especially that surrounding the web and node, is likely to be hit the hardest. The more driven the open source project was by its proximity to either an employed contributor or actively dependent business, the bigger the impact from a shift to language models will be.

This is a serious problem for the software industry as arguably much of the economic value the industry has provided over the past decade comes from strip-mining open source and free software.

6. Licence contamination

Microsoft and Google don’t train their language models on their own code. GitHub’s Copilot isn’t trained on code from Microsoft’s office suite, even though many of its products are likely to be some of the largest React Native projects in existence. There aren’t many C++ code bases as big as Windows. Google’s repository is probably one of the biggest collection of python and java code you can find.

They don’t seem to use it for training, but instead train on collections of open source code that contain both permissive and copyleft licences.

Copyleft licences, if used, force you to release your own project under their licence. Many of them, even non-copyleft, have patent clauses, which is poison for quite a few employers. Even permissive licences require attribution, and you can absolutely get sued if you’re caught copying open source code without attribution.

Remember our Chekhov’s gun?

👉🏻👉🏻 memorisation!

Well, 👉🏻👉🏻 pewpew!!!

Turns out blindly copying open source code is problematic. Whodathunkit?

These models all memorise a lot, and they tend to copy what they memorise into their output. GitHub’s own numbers peg verbatim copies of code that’s at least 150 characters at 1%^[9], which is roughly the same, in terms of verbatim copying, as what you seem to get in other language models.

For context, that means that if you use a language model for development, a copilot or chatbot, three or four times a day, you’re going to get a verbatim copy of open source code injected into your project about once a month. If every team member uses one, then multiply that by the size of the team.

GitHub’s Copilot has a feature that lets you block verbatim copies. This obviously requires both a check, which slows the result down, and it will throw out a bunch of useful results, making the language model less useful. It’s already not as useful as it’s made out to be and pretty darn slow so many people are going to turn off the “please don’t plagiarise” checkbox.

But even GitHub’s checks are insufficient. The keyword there is “verbatim”, because language models have a tendency to rephrase their output. If GitHub Copilot copies a GPLed implementation of an algorithm into your project but changes all the variable names, Copilot won’t detect it, it’ll still be plagiarism and the copied code is still under the GPL. This isn’t unlikely as this is how language models work. Memorisation and then copying with light rephrasing is what they do.

Training the system only on permissively licensed code doesn’t solve the problem. It won’t force your project to adopt an MIT licence or anything like that, but you can still be sued if it’s discovered.

This would seem to give Microsoft and GitHub a good reason not to train on the Office code base, for example. If they did, there’s a good chance that a prompt to generate DOCX parsing code might “generate” a verbatim copy of the DOCX parsing code from Microsoft Word.

And they can’t have that, can they? This would both undercut their own strategic advantage, and it would break the illusion that these systems are generating novel code from scratch.

This should make it clear that what they’re actually doing is strip-mine the free and open source software ecosystem.

How much of a problem is this?

—It won’t matter. I won’t get caught.

You personally won’t get caught, but your employer might, and Intellectual Property scans or similar code audits tend to come up at the absolute worst moments in the history of any given organisation:

• During due diligence for an acquisition. Could cost the company and managers a fortune.
• In discovery for an unrelated lawsuit. Again, could cost the company a fortune.
• During hacks and other security incidents. Could. Cost. A. Fortune.

“AI” vendors won’t take any responsibility for this risk. I doubt your business insurance covers “automated language model plagiarism” lawsuits.

Language models for software development are a lawsuit waiting to happen.

Unless they are completely reinvented from scratch, language model code generators are, in my opinion, unsuitable for anything except for prototypes and throwaway projects.

So, obviously, everybody’s going to use them

• All the potentially bad stuff happens later. Unlikely to affect your bonuses or employment.
• It’ll be years before the first licence contamination lawsuits happen.
• Most employees will be long gone before anybody realises just how much of a bad idea it was.
• But you’ll still get that nice “AI” bump in the stock market.

What all of these problems have in common is that their impact is delayed and most of them will only appear in the form of increased frequency of bugs and other defects and general project chaos.

The biggest issue, licence contamination, will likely take years before it starts to hit the industry, and is likely to be mitigated by virtue of the fact that many of the heaviest users of “AI”-generated code will have folded due to general mismanagement long before anybody cares enough to check their code.

If you were ever wondering if we, as an industry, were capable of coming up with a systemic issue to rival the Y2K bug in scale and stupidity? Well, here you go.

You can start using a language model, get the stock market bump, present the short term increase in volume as productivity, and be long gone before anybody connects the dots between language model use and the jump in defects.

Even if you purposefully tried to come up with a technology that played directly into and magnified the software industry’s dysfunctions you wouldn’t be able to come up with anything as perfectly imperfect as these language models.

It’s nonsense without consequence.

Counterproductive novelty that you can indulge in without harming your career.

It might even do your career some good. Show that you’re embracing the future.

But…

The best is yet to come

In a few years’ time, once the effects of the “AI” bubble finally dissipates…

Somebody’s going to get paid to fix the crap it left behind.

The best way to support this newsletter or my blog is to buy one of my books, The Intelligence Illusion: a practical guide to the business risks of Generative AI or Out of the Software Crisis. Or, you can buy them both as a bundle.

How inputs are supposed to work

Inputs are a necessary part of computing. To get stuff out, you need to get stuff in.

As important is our need to control how the computer interprets that input.

There’s a limit to how useful it is to just move numbers around in the machine, at some point you’re going to want to do something with the numbers.

Ultimately all input gets processed by code, but computers wouldn’t be much of a productivity boon if you had to program every single task by hand.

That’s why we have controls that are separate from input. On the command line, every tool has standard input so you can pipe text into it. But it also has commands and structured ways of controlling how the tool handles the input that are separate from the input itself.

Most protocols have something similar. HTTP separates headers from the body of the message. The headers indicate how the body should be handled.

SQL, the language that’s almost universally used in software development to interact with databases, lets you separate the input, or parameters, from the control statement itself through binding.

You only include named placeholders for the input data in the statement and then prepare the statement for use by binding the actual values to the placeholders separately. That way they can’t affect the statement itself.

Historically, SQL is itself a good example of what happens when you don’t separate input from control: disaster.

It used to be the norm to just blob everything into the SQL statement itself and just lob it over into your database like a grenade.

This gave the input full access to the database. They could exfiltrate data, change anything, delete anything—it gave a random attacker the power to do anything the server sending the statement could normally do.

This is, unfortunately still an issue in web software. As an industry software is not much for learning or improving. We tend to focus on finding new ways of making old mistakes.

How language model prompts work

A good example of this is how the industry is integrating language models into their software. Prompts are the only real control surface that language models have.

They are also their only standard input. All in one blobby, undifferentiated jumble. Just like old-style SQL.

Every integration, so far, of a language model with a larger production system involved jamming the control prompt, provided by the developer, and the input, provided by the end-user, together and schlepping it over to the language model that interprets it as a single text.

The control prompt usually included language that tells the model not to listen to control statements in the input, but because it’s all input into the model as one big slop, there’s nothing really to prevent an adversarial end-user from finding ways to countermand the commands in the developer portion of the prompt.

The AI industry calls this jailbreaking, but it’s honestly the incarceration equivalent of an inmate being let out after showing a note to a guard where they had scrawled: “Inmate haz pardon. Pliz let out. Signed, Presedent!”

That can’t be true, right? That sounds like a joke? This is the next big thing in computing! We must have come up with some realistic mitigations this time?

Right?

Unfortunately, no.

We’ve known about this issue from the beginning of ChatGPT:

Simon Willison, in particular, has done a good job of documenting the issue:

• Prompt injection attacks against GPT-3
• I don’t know how to solve prompt injection
• You can’t solve AI security problems with more AI
• Prompt injection: What’s the worst that can happen?
• Prompt injection explained, with video, slides, and a transcript
• Delimiters won’t save you from prompt injection

That last one is worth noting. The solution to prompt injections proposed by OpenAI doesn’t work.

This, in and of itself, should be a surprise. OpenAI is a research company that has been thrust into the consumer software business. They have no competency in software or system security.

That may sound harsh, but they’ve already had a few security-related incidents:

• OpenAI admits some premium users’ payment info was exposed. Whoops.
• OpenAI Confirms Leak of ChatGPT Conversation Histories. Eh, it’s not as if people are pasting confidential information into ChatGPT, right?
• 11% of data employees paste into ChatGPT is confidential. I’m sure this is fine. I mean, the rest of tech has faith in them, right?
• Big Tech is already warning us about AI privacy problems. For Apple, Samsung, Verizon, and a host of banks “chatGPT has been on the ban list for months”. Welp.

OpenAI doesn’t have an organisation that prioritises or understands software security. Whatever security they have is all patched on—jumbled in like shake-and-bake.

It’s unlikely that we’ll see a realistic and pragmatic solution to prompt injections from OpenAI, which is a problem because both their models and methods are closed, proprietary, and hidden from scrutiny by independent researchers and regulators.

You know researchers and regulators.

They’re the people who usually make sure that software vulnerabilities don’t get out of hand and take down an entire computing ecosystem. They’re the ones who discover unpatched vulnerabilities that companies were trying to pray away even as attackers all over the world are exploiting them.

Y’know, the people on our side.

At the same time all of this is not going on, OpenAI and others are just working so hard at expanding the potential attack surface of these systems.

Who has time to fix vulnerabilities?

They sure don’t.

Now they’re giving us plugins, all the better to let attackers extend their reach into external and internal services. They’re giving us APIs, so you can make sure your own products have prompt injection vulnerabilities. And their partners are giving us prompt-controlled customer service bots, all the better for users to—I don’t know—use their full access to your systems to fix their problems themselves, I guess.

The AI industry is doing it, so it must be a good idea, right?

Right.

Prompts—and with them language models—are not fit for purpose

They are not safe. They are not secure. Language models should neither be integrated with external services nor should they be exposed to external users. You absolutely should not do both.

I know of a couple of ecommerce stores in Iceland that had open SQL injection vulnerabilities for over a decade. They’re fixed now, and they were lucky this never became a disaster.

Maybe you’ll be lucky as well.

All I know is a lot of companies and people are right now using extremely unsafe methods to integrate language models into their products and services. Their only real defence is luck.

And, relying on luck is, generally speaking, a bad strategy for software security.

The best way to support this newsletter or my blog is to buy one of my books, The Intelligence Illusion: a practical guide to the business risks of Generative AI or Out of the Software Crisis.

The EU AI Act is still subject to change, although most people don’t expect too many major changes at this point. Any of this could change.

Also, IANAL. I don’t even play one on TV.

• Safe-harbour provisions for service provider liability are unaffected by the EU’s AI Act. Hosting rules are unaffected.
• Developers (not deployers) of foundation models need to register their models, with documentation, prior to making it available on the market or as a service.
• Foundation models need to come with documentation about their training data set and pass a number of to-be-implemented standardised benchmarks that examine the suitability of the data they use in terms of biases and other factors.
• The developers of a foundation model are responsible for compliance, not the deployers.
• Providers of Generative AI systems are required to document and publish detailed summaries of the copyright-protected training data they used, as a part of the registration process.
• The Act is clearly designed to benefit AI research through increased transparency and documentation.
• It bans a bunch of things that shouldn’t have been allowed in the first place.
• If you take a foundation model, fine-tune it for a specialised purpose, and deploy it as a part of your software, it won’t count as a foundation model, and you’ll probably be fine, as long as the original provider of the foundation model was compliant.
• If you’re using a foundation model over an API to add a specialised feature to your software, then you’ll probably be fine, as long as the original developer was compliant.

The AI Act covers a lot. It covers the use of AI for biometric identification, high-risk systems whose intended purpose involves people’s health and safety (or life and liberty), foundation models, generative AI, and your run-of-the-mill AI/ML software. It’s also painfully aware that these are early days and that regulators need to be flexible.

The focus of this essay is just foundation models and generative AI, and even with that narrow focus it’s already much too long.

The AI industry is having a temper tantrum

If you’ve been paying attention to tech social media over the past few days, you’ll have seen the outcry about the EU’s proposed AI Act.

The act isn’t final. It’s still subject to negotiation between various parts of the EU infrastructure and how it gets implemented can also change its effect in substantial ways.

That isn’t preventing the US tech industry from panicking. In a blog post that was later popularised by a noted tech commentator, AI enthusiasts have claimed that the EU is doing several very bad, double-plus ungood things and, with it, we Europeans are dooming ourselves to something or the other:

• They’re banning open source AI models!
• It’ll be illegal to host AI models or code!
• They’re banning AI models accessed via an API.
• They’re banning fine-tuning of foundation models!

I’ve struck out the statements in the list above because, unfortunately for those who like a good panic, none of them seem to be true. With the act and the recent actions by GDPR regulators, the EU has joined AI ethicists such as Emily M. Bender, Timnit Gebru, and others on the tech industry’s Enemies of AI list.

The crimes of the ethicists, according to tech:

• A refusal to believe in an unfounded expectation of endless exponential growth.
• An insistence that models be evaluated based on genuine, not imagined, functionality.
• The clearly irrational belief that AI development should be transparent, sustainable, and avoid harming the societies we live in.

The EU’s crimes:

• A hatred of innovation and the future.
• An insistence on legislating themselves into the stone age.
• A completely irrational disbelief in the wonders provided so generously by the glorious, kind, and all-around awesome people in the tech industry.

Or, something.

It’s hard to keep track of industry and investor consensus now that bubble mania has set it, especially since quite a few of them are so helpfully using ChatGPT to generate fact-free incoherence for them.

(Imagine a meme of a greying scruffy dog turning its head to one side and going “roo?”. That’s me trying to parse some of the social media posts coming from AI fans. Most of it’s just “what?”)

I’m going to ignore the tantrums and instead have a look, for myself, at what the current proposal for the Act says. For this I’m using the consolidated PDF document of the amended act as published by the European Parliament as a reference.

Scope and service provider liability

Right at the outset of the act in Article 2: Scope, it makes it clear that it doesn’t intend to override existing safe-harbour laws for service providers:

5. This Regulation shall not affect the application of the provisions on the liability of intermediary service providers set out in Chapter II, Section IV of Directive 2000/31/EC of the European Parliament and of the Council6 [as to be replaced by the corresponding provisions of the Digital Services Act].

5b. This Regulation is without prejudice to the rules laid down by other Union legal acts related to consumer protection and product safety.

5c. This Regulation shall not preclude Member States or the Union from maintaining or introducing laws, regulations or administrative provisions which are more favourable to workers in terms of protecting their rights in respect of the use of AI systems by employers, or to encourage or allow the application of collective agreements which are more favourable to workers.

5d. This Regulation shall not apply to research, testing and development activities regarding an AI system prior to this system being placed on the market or put into service, provided that these activities are conducted respecting fundamental rights and the applicable Union law. The testing in real world conditions shall not be covered by this exemption. The Commission is empowered to may adopt delegated acts in accordance with Article 73 to specify this exemption to prevent its existing and potential abuse. The AI Office shall provide guidance on the governance of research and development pursuant to Article 56, also aiming at coordinating its application by the national supervisory authorities.

5d. This Regulation shall not apply to AI components provided under free and opensource licences except to the extent they are placed on the market or put into service by a provider as part of a high-risk AI system or of an AI system that falls under Title II or IV. This exemption shall not apply to foundation models as defined in Art 3.

The first and most important part here is clause 5.

“Chapter II, Section IV of Directive 2000/31/EC” is the EU’s version of Section 230 that governs “liability of intermediary service providers”. It covers hosting, “mere conduit” providers, caching, and forbids member states from imposing a general obligation to monitor on service providers. The AI Act specifically says that it does not affect the liability of intermediate service providers.

This means that, yes, GitHub and other code repositories are still allowed to host AI model code. Hosting providers don’t have any additional liability under the AI Act, only the providers of the models themselves and those who deploy them.

Existing rules about hosting still apply. Same as it’s been for the past twenty-three years.

Clauses 5d are probably the source of some of the tech industry’s confusion and anger. I’m guessing they interpret (or ChatGPT interpreted for them) the “this exemption shall not apply to foundation models” as applying to all the clauses from 5 to 5d, so they assume that none of those exceptions apply to foundation models, which would mean that the safe-harbour provision is indeed overridden.

That interpretation makes no sense because that would also mean that clauses 5b and 5c would also get dropped

5c in particular is about the EU reserving the right of member states to introduce further laws to protect labour from employers abusing AI software.

I can guarantee you that the Act isn’t intended to prevent the EU from making further legislation on foundation models.

The EU is also quite fond of it’s consumer protection laws and wouldn’t give foundation models a pass on those.

This means that interpreting “shall not apply to foundation models” as applying to all the exceptions is almost certainly nonsense.

There’s also a chance that people in the tech industry think that Article 10, which sets out strict data governance rules, applies to foundation models, but that article is in Chapter 2: Requirements for high-risk AI systems.

The act makes it clear that “foundation” and “high-risk” are two distinct categories and that articles 8-15 apply to high-risk systems and not foundation models and that their obligations are separate (p. 143).

For high-risk AI systems, the general principles are translated into and complied with by providers or deployers by means of the requirements set out in Articles 8 to 15, and respective obligations laid down in Chapter 3 of Title III of this Regulation. For foundation models, the general principles are translated into and complied with by providers by means of the requirements set out in Articles 28 to 28b.

And from page 29:

These specific requirements and obligations do not amount to considering foundation models as high risk AI systems.

What 5d means is that the pre-release development of foundation models has to follow the rules set out in the regulation on foundation models.

That would seem to mean that there are requirements for foundation models that you need to follow during model training in addition to those that come into effect once you put it into service, which is when the regulation kicks in for other AI models.

That very much isn’t a ban of any kind, but maybe the rules and requirements are onerous? Maybe that’s why the panic?

But first, what do the words mean?

We need to find out what the EU AI Act means with things like “foundation model”, “provider”, and “deployer”.

From Article 3:

(1c) ‘foundation model’ means an AI model that is trained on broad data at scale, is designed for generality of output, and can be adapted to a wide range of distinctive tasks;

That seems to match the industry’s definition of the term. You could quibble that this is a bad way of describing these models in the first place, but that’s generally not a debate that EU regulators are going to get involved in. As far as I can tell, they usually prefer to reuse industry terms, possibly with a little more specificity, when they can.

Also, from page 28:

Pretrained models developed for a narrower, less general, more limited set of applications that cannot be adapted for a wide range of tasks such as simple multipurpose AI systems should not be considered foundation models for the purposes of this Regulation, because of their greater interpretability which makes their behaviour less unpredictable.

That lets many fine-tuned models off the hook.

Back to Article 3:

(23) ‘substantial modification’ means a modification or a series of modifications of the AI system after its placing on the market or putting into service which is not foreseen or planned in the initial risk assessment by the provider and as a result of which the compliance of the AI system with the requirements set out in Title III, Chapter 2 of this Regulation is affected or results in a modification to the intended purpose for which the AI system has been assessed

This is an important note because model types that need to be registered (high-risk and foundation) also need to be re-registered after every substantial modification, which some have interpreted as a ban on a variety of approaches to ongoing model improvement. This explains that these methods for ongoing fine-tuning or learning do not force you to re-register the model, because those modifications are foreseen. The same thing applies to security updates and modifications geared towards the ongoing mitigation of misuse and bias.

If you’re familiar with semantic versioning, you probably only need to register major versions.

(2) ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or that has an AI system developed with a view to placing it on the market or putting it into service under its own name or trademark, whether for payment or free of charge;

“Provider” seems to mean whichever legal entity is developing an AI system, which doesn’t necessarily have to be the same entity as the one who deploys it. “Placing on the market” in this context means the EU market. You can alpha- or beta-test non-foundation models on US customers as much as you like and the EU won’t care.

(4) ‘deployer’ means any natural or legal person, public authority, agency or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity.

Most of the requirements the EU sets are on providers not deployers. If the foundation model is compliant and registered, then the organisations who deploy and use them should be fine.

The rules everybody has to follow

The act sets out general principles all AI models should follow—that providers should “make their best efforts” to follow.

They all seem innocuous (from p. 143):

1. “AI systems shall be developed and used as a tool that serves people, respects human dignity and personal autonomy, and that is functioning in a way that can be appropriately controlled and overseen by humans.”
2. “AI systems shall be developed and used in a way to minimize unintended and unexpected harm as well as being robust in case of unintended problems and being resilient against attempts to alter the use or performance of the AI system so as to allow unlawful use by malicious third parties.”
3. “AI systems shall be developed and used in compliance with existing privacy and data protection rules, while processing data that meets high standards in terms of quality and integrity.”
4. “AI systems shall be developed and used in a way that allows appropriate traceability and explainability, while making humans aware that they communicate or interact with an AI system as well as duly informing users of the capabilities and limitations of that AI system and affected persons about their rights.”
5. “AI systems shall be developed and used in a way that includes diverse actors and promotes equal access, gender equality and cultural diversity, while avoiding discriminatory impacts and unfair biases that are prohibited by Union or national law.”
6. “AI systems shall be developed and used in a sustainable and environmentally friendly manner as well as in a way to benefit all human beings, while monitoring and assessing the long-term impacts on the individual, society and democracy.”

Maybe I’m wrong, but none of this looks like world-ending stuff, and most of it is fairly close to what you’re seeing regulators in other territories talk about. At least the ones that haven’t fallen for the AGI sci-fi nonsense the industry is peddling.

The notable bit is the requirement that users should be properly informed when they’re interacting with an AI system. This comes up again in Article 52: Transparency obligations for certain AI systems and repeated in the foundation model requirements, which would seem to indicate that EU regulators consider informed consent by the end-user to be rather quite important.

The word “appropriate” is used in the other two clauses that are genuinely AI specific, which is going to be implementation-specific, based largely on researcher and industry feedback, and likely make them pretty close to toothless in practice. The rest is vague enough to boil down to “please follow existing regulations and laws, even though you think AI should be exempt because it’s so cool”.

You also have a list of prohibited practices that are set out in Article 5. Those boil down to:

• Subliminal manipulation or intentionally distorting human behaviour in a material way that’s likely to cause harm.
• It’ll ban phrenological-style applications such as does a person with this skull shape do crime?. These have been popular with law enforcement. Say goodbye to “this AI detects homosexuality” or “this AI detects sociopathy” kind of pseudoscientific nonsense systems.
• Specifically targeting vulnerable sections of the population.
• Social credit scoring.
• Real-time biometric identification of people in public spaces.
• Untargeted scraping of facial images for the purposes of expanding facial recognition databases.

None of those seem to apply to foundation or generative models, but you can already tell why the tech industry hates this proposed act. This is like a list of all their favourite things. Banning phrenology is to AI industry investors about as evil as shoving kittens into a meat grinder.

To sleazy VC types it’s like setting a law that bans puppy dogs and rainbows.

The foundation model requirements

At last, we’re getting to the proper stuff. Foundations models, what’s it all about?

The requirements specific to the providers of foundation models are outlined in Article 28b on pages 39–41 of the document linked to by the European Parliament news item.

1. The first requirement is just basic risk assessment and mitigation. “Demonstrate through appropriate design, testing and analysis that the identification, the reduction and mitigation of reasonably foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law prior and throughout development with appropriate methods such as with the involvement of independent experts, as well as the documentation of remaining non-mitigable risks after development.”
2. “Process and incorporate only datasets that are subject to appropriate data governance measures for foundation models, in particular measures to examine the suitability of the data sources and possible biases and appropriate mitigation;”
3. “Design and develop the foundation model in order to achieve throughout its lifecycle appropriate levels of performance, predictability, interpretability, corrigibility, safety and cybersecurity.”
4. “Design and develop the foundation model, making use of applicable standards to reduce energy use, resource use and waste, as well as to increase energy efficiency, and the overall efficiency of the system.”
5. “Draw up extensive technical documentation and intelligible instructions for use in order to enable the downstream providers to comply with their obligations.”
6. “Establish a quality management system to ensure and document compliance with this Article, with the possibility to experiment in fulfilling this requirement”
7. “Register that foundation model in the EU database referred to in Article 60.”

If I’m to be brutally honest, except for registration, the requirements above are basically what you were supposed to do when selling a large-scale machine learning system to an enterprise or institution a short while ago, before bubble-mania kicked in. It’s the sort of stuff you should be doing anyway.

The only difference here is that OpenAI, Microsoft, and Google now all think it’s a strategic advantage to keep all of it secret, even though that secrecy directly threatens the viability of AI research and cripples the ability of their customers to assess and plan around the limitations of their products.

Forcing AI vendors to publish this information is an obvious benefit to all of us, even them, because their AI systems are the product of AI research and the secrecy they are currently employing risks turning the entire field into a dead end, and trigger another “AI winter”.

More importantly, forcing them to gather this data and documentation and making it available to others is only going to be a benefit for the AI industry in general, in the long term.

Just look at how quickly open source developers managed to replicate the approaches and strategies from Facebook’s LLaMA model. This would be like that, just on steroids.

This is exciting, not scary.

The act also has a requirement where the EU AI office, in collaboration with international partners, has to “develop cost-effective guidance and capabilities to measure and benchmark aspects of AI systems and AI components, and notably of foundation models relevant to the compliance and enforcement of this Regulation based on the generally acknowledged state of the art” (p. 92).

This puts the onus on the EU to provide straightforward benchmarks—most likely based on existing benchmarks in AI research (see “generally acknowledged state of the art”) or organisations like Hugging Face—that providers can use when developing foundation models.

Given that most providers of existing foundation models are already using benchmarks to guide their development work, and that they’ll almost certainly have a say in the development of these benchmarks, this doesn’t seem that problematic.

In fact, you could argue that it’s much too lax considering the misbehaviour of the tech industry over that past decade.

AI ethicists to the rescue

The AI industry loves to hate AI ethicists.

The doomers have a purpose: they are—in effect—constantly talking up the capabilities of these systems and the “geniuses” who make them.

But, ethicists? Focusing on existing, not hypothetical, harms? Insistent on talking about models in terms of their genuine, not imagined, capabilities? Transparency? Consent?

Ugh.

Clearly, these are some very bad people who just hate technology.

What the AI industry and hangers-on are missing is that AI ethicists are possibly the most constructive force in the field of AI research today.

Anybody who is trying to stop you from getting behind the wheel of a car when you’re drunk is your friend, not your enemy.

The industry today is vastly over-promising on the capabilities of their AI systems. They are shipping them without any meaningful safeguards or acknowledgement of how they’re harming our digital commons, creative industries, minorities, or how they are the perfect tool for misinformation at scale.

The risk is enormous and directly threaten the AI vendors themselves. Universal misinformation and a collapsed digital commons is an existential threat to a search engine. The creative industries are some of the biggest software customers around—replacing million dollar customers with twenty dollar customers is just bad business. Language and diffusion model abuses harm tech companies just as much in the long term as it does the rest of us.

The people warning you to not make these mistakes are your allies. They’re fighting in your corner, but you keep punching them in the back.

What’s more, they’re likely to save you—or at least open source models—from the EU AI Act by making compliance as good as automatic.

The documentation requirements might seem onerous, but documentation and transparency is also a hard requirement for the advancement of AI research in general, so it shouldn’t come as a surprise that a lot of work has already been done.

AI researchers, ethicists, and Hugging Face in particular have accomplished a lot, with more no doubt on the way.

Their work includes:

• Margaret Mitchell with Model Cards for Model Reporting implemented as Model Cards at Hugging Face.
• Emily M. Bender and Batya Friedman with Data Statements for Natural Language Processing.
• Timnit Gebru et al. with Datasheets for Datasets.
• Julia Stoyanovich and Bill Howe with Nutritional Labels for Data and Models
• Kasia S. Chmielinski, Sarah Newman, et al. with The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence
• Hugging Face have shipped a tool with the goal of at least partially automating EU AI Act compliance checks: Model Card Regulatory Check or RegCheck AI.
• Hugging Face, in particular, seems to have broadly good intentions: What does ethical AI look like?

AI researchers have been preparing for this for years because, as I said, documentation and transparency is essential for the field to progress.

It’s OpenAI, Microsoft, and Google who are holding the industry back with their secrecy and risk-taking.

It seems more likely than not that major open source language models will be broadly compliant with the EU AI Act well before the act takes effect. Researchers are setting standards and processes for gathering and presenting the documentation and the ethics team at Hugging Face seems to be putting it into practice.

Seriously, in terms of reducing the cost of regulatory compliance for the industry, and in terms of broadly increasing quality through assessment of bias and functionality, the value that AI ethicists are creating for the industry is enormous. Open source models likely won’t be viable for serious use without them.

All the industry accomplishes by demonising this group is increase their future liabilities and reduce the long term value of their products.

But, I saved the best for last

There is one additional set of requirements for generative foundation models (p. 41). They need to:

1. Comply with transparency requirements, making sure that generative output is correctly labelled, similar to what Adobe is already planning to do with their generative image output.
2. Prevent content that’s illegal in the EU, such as child abuse imagery.
3. Make available a “detailed summary of the use of training data protected under copyright law.”

This is where most existing proprietary foundation models would end up getting banned in the EU.

The labelling requirement is fairly straightforward and is a requirement that’s likely to be echoed in many other jurisdictions, anyway.

The illegal content requirements aren’t that much of an issue as most existing providers try to prevent that kind of output, anyway.

Open source models will be just fine on the “detailed summary” front as their training data set isn’t a secret.

But GPT-4 and PaLM? Yeah, the third requirement is where they both get stomped. Not because they can’t. It’s highly likely that Google and Microsoft have more than enough documentation to provide a detailed summary of the copyright-protected training data they used. If they don’t, then they are incompetent and should get stomped hard, then investigated and fined.

They probably have that documentation somewhere. They just really don’t want to publish it because it’s almost certainly all copyright-protected material. Whatever public domain or freely licensed data they’ve used is only going to be a small part of the big models.

The data belongs to others, many of whom also happen to be directly threatened by OpenAI and Google introducing generative AI, or at the very least will want their cut of the AI bubble pie.

That’s a recipe for an avalanche of major lawsuits from big copyright-holding corporations.

That’s why you’re going to hear a lot of scaremongering about the EU AI Act from all over the tech industry.

The best way to support this newsletter or my blog is to buy one of my books, The Intelligence Illusion: a practical guide to the business risks of Generative AI or Out of the Software Crisis.

In the ancient days, when mammoths prowled, search engines trusted the text in the page

I’m old enough to not only remember what the web was like before Google, I remember what it was like before AltaVista, which was the Google before Google.

For those of you who aren’t internet ancients, in the early days of the web finding things was made easy by the fact that there just weren’t that many websites. You could list most websites in a manageable, human-curated directory. When I made my first website, a collection of essays on comics as literature (what can I say, I’ve always been a nerd), I submitted it to a few online directories and overnight it got traffic. I got my first email from a reader the next morning.

But, the web continued to explode in popularity, and before long the directories weren’t handling the sheer volume. People started looking for search engines—no, demanding search engines. The web wasn’t working without one.

Quite a few companies tried, but the first one to deliver what looked like a solid experience as well as decent quality results was AltaVista. Everybody loved it. Everybody switched to it. For a while.

But it had a fatal flaw: it trusted the text in the page. Not only did it trust the text on the page, it was the primary factor it used in deciding whether the page was relevant to the query or not. A key part of that was the infamous “meta keyword tag”. Developers today know meta tags as a fairly innocuous, if awkward, method for injecting metadata into pages. Services then use it for previews and the like. But back in the day, what you had in the meta keyword tag decided where your page landed in the search engine results.

• It didn’t matter how many other pages linked to it.
• The credibility of the domain, or lack thereof, wasn’t taken into account.
• The structure and semantics of the page, while not ignored, were largely irrelevant to AltaVista.

No, it was all down to the meta tag, so every sleazy marketdroid on the web stuffed theirs. AltaVista’s search results were filled with irrelevant content. Or, no content, when they clamped down on keyword-stuffing, because it turns out that even then, the web was dominated by sleazy marketdroids.

Trusting and prioritising the meta tag was a security vulnerability of sorts, one that Google avoided from the start, and AltaVista only dropped in 2002. As a Search Engine Watch article explained when AltaVista dropped their “support”:

The first major crawler-based search engines to use the meta keywords tag were Infoseek and AltaVista. It’s unclear which one provided support first, but both were offering it in early 1996. When Inktomi launched in mid-1996 through the HotBot search engine, it also provided support for the tag. Lycos did the same in mid-1997, taking support up to four out of the seven major crawlers at the time (Excite, WebCrawler and Northern Light did not provide support).

The ascendancy of the tag did not last after 1997. Experience with the tag has showed it to be a spam magnet. Some web site owners would insert misleading words about their pages or use excessive repetition of words in hopes of tricking the crawlers about relevancy. For this reason, Excite (which also owned WebCrawler) resisted added support. Lycos quietly dropped its support of the tag in 1998, and newer search engines such as Google and FAST never added support at all.

After Infoseek (Go.com) closed in 2000, the meta keywords tag was left with only two major supporters: AltaVista and Inktomi. Now Inktomi remains the only one, with AltaVista having dropped its support in July, the company says.

Why does this matter today? Surely, nobody would be dumb enough to build an information management system that is so utterly, completely open to keyword manipulation?

Well…

Language models are to modern search what the meta tag was to AltaVista

Last week I wrote about The Poisoning of ChatGPT and how researchers had, in recent years, discovered that language models can be poisoned through their training data—both the data used in the initial training and fine-tuning.

The researchers managed to do both keyword manipulation and degrade output with as few as a hundred toxic entries, and they discover that large models are less stable and more vulnerable to poisoning. They also discovered that preventing these attacks is extremely difficult, if not realistically impossible.

Of course, because they are AI researchers and the entire field has fundamental issues with finding accurate names for complex topics, the industry has decided to call these attacks poisonings when most of the poisoning attacks they outline are more properly keyword manipulation exploits.

You know… literally the job description of a black-hat SEO.

Moreover, researchers have also discovered that it’s probably mathematically impossible to secure the training data for a large language model like GPT-4 or PaLM 2. This was outlined in a research paper that Google themselves tried to censor, an act that eventually led the Google-employed author, El Mahdi El Mhamdi, to leave the company. The paper has now been updated to say what the authors wanted it to say all along, and it’s a doozy.

This paper emphasized three characteristics of the data on which LAIMs are trained. Namely, they are mostly user-generated, very high-dimensional and heterogeneous. Unfortunately, the current literature on secure learning, which we reviewed, shows that these features make LAIMs inherently vulnerable to privacy and poisoning attacks. Large AI models are bound to be dangerous. Their rushed deployment, especially at scale, poses a serious threat to justice, public health and to national and international security.

The only realistic way to defend against poisoning is to use stale training data. As soon as you start to include fresh pages in a data set this large, you de facto lose the ability to defend the integrity of the data set and with it the integrity of the language model’s output.

Major language model vendors, such as OpenAI, have decided to sacrifice “freshness” in order to preserve what little integrity their systems have in the first place—remember hallucinations are still an unsolved problem.

Except Google. They have decided that “freshness” is in their corporate DNA. They want to be up-to-date at all costs, so their training data now goes all the way up to February 2023, and I have no doubt they plan on keeping it as “fresh” as possible, with each update likely bring on ever multiplying attempts to manipulate their model.

Google is rushing ahead to “catch up” on AI without paying any attention to the security or integrity of its products, something that its own employees, past and present, have been warning it about.

They are ignoring the acute vulnerability that large language models have with keyword manipulation exploits, making them the modern equivalent of the search engines of the 90s. The only thing that’s different today is that there is now much more money in manipulating search engines than ever before, which makes the vulnerability of large language models a lethal issue for search, research, or information management at scale.

But Google doesn’t care because they want that AI stock price bump. That’s all that matters. They don’t even see how they’re marching down the same road that AltaVista went down twenty-five years ago.

If we’re lucky, Google Bard will flop and none of this will ever become an issue.

But, if we’re really unlucky, then the future of search is LLMs and rampant keyword manipulation.

The best way to support this newsletter or my blog is to buy one of my books, The Intelligence Illusion: a practical guide to the business risks of Generative AI or Out of the Software Crisis.