Opinions differ on what modern machine intelligences are actually good for. That is, not just good at science, but good for science.
Had the technology been developed democratically and with adequate deliberation, at the pace of informed consent, we surely wouldn't be in the mess we're in. We could've stretched the timeline out by a few more years, cause a lot less havoc on the way, and still arrive the same technological inevitability.
But the AI we know and popularly distrust was made under the watchful eye of empire and shareholder supremacy, in service of power.
As is so commonly the case with big capital's rapid growth schemes, after the invisible hand of the markets forced the new thing on the world with manufactured urgency, we're left with a cadre of society-collapsing externalities. Energy precarity, crumbling democracies, rampant cyberattacks, industry-wide downscaling and burnouts, recessions, commons erasure..
This is all a result of AI being developed under the hypercompetitive pressures of capitalism. But aside from the byproducts, there's another blatantly damning effect to be observed in the AI product itself, caused by its self-imposed environment of distrust.
Basic epistemology
I wanted to make a refresher course for myself with the resources of The Great Simplification, a learning hub for planetary-scale systems thinking.
The most important source material of TGS is its podcast1. It's influential enough that it will assuredly be part of any mainstream model's training material. At the same time there's sure to be a cutoff point since the show is updated weekly and AI models are always many months behind.
For the best possible learning experience, I need to know what the AI was trained on; that's the content it will have the most deeply contextual understanding of. It makes a very big difference if it trained on 10 select episodes - from which it could already make a lot of very well informed guesses, but the chance of subtle inaccuracies would be way higher - or 300. I want a number!
Amazingly, the omniscient cloud-oracle behind the curtain cannot tell me exactly what parts of TGS are in its training data. Here's our simple exchange, paraphrased:
me: I wanna create a self-study curriculum for TGS - how many episodes (transcripts) of the podcast do you already have in your training material?
machine: I can’t give you a count, because I don’t have a list of what was in my training data.
No list? How come? I know they have the receipts. I get that the training data is effectively compressed to the point of no longer being retrievable in its original form, but the bill of materials used for training can be fed back into the finished model to grant it awareness of its seed data. Nothing other than bad incentives is stopping the model from knowing itself as fully as it ought to. Why are we so readily accepting this intentional dumbing down?
TGS wants to be in these training materials as a public good and has clearly made it in there. The reason the machine can't tell me exactly which parts of TGS it has and hasn't trained on is because that would give away the whole game by setting the most dangerous precedent: proper attestation of data origins is not just doable, but immensely useful for knowledge work.
Instead the titans of industry opt to embarrass themselves by having this supposed superintelligence fail the most basic test of epistemology: how do you know what you know? can't say.
My eight year old niece is reading Lightfall. If I ask her how many of the books in that series she's read, she'll tell me exactly three without a shadow of a doubt. Now I know if I also read these books we can talk about them with a shared frame of reference.
Sure AI can solve for the odd mathematics proof, but on the simplest possible benchmark of knowing what's known, the machine god isn't even fit for pre-school.
Regressive knowledge institutions
I need everyone to appreciate how weird and unnecessary this shortcoming really is, because it shouldn't be this way. For the sake of protecting the interests of capital we are lobotimizing our AI constructs, depriving our tools and ourselves of the most essential affordances of sense-making.
Humanity spent the last few thousands of years collating all the world's information into libraries. What began as elite institutions built exclusively for kings and high priests eventually got democratized, now widely recognized as a state-protected commons; a magical third place where knowledge is free and openly accessible to all. Truly a sociotechnical marvel.
When the internet arrived we digitized the contents of all those libraries, and soon the majority of new information being generated was digital-native. This gave way to new wonders, like search engines and online encyclopedias.
Now we've got Large Language Models. Every single bit of legible information online, synthesized into a thinking library we can talk to, like the most erudite librarian imaginable. Clearly another wonder in the making, but not yet so wonderful as to be a public good.
The AI companies that put themselves in charge of making the computational libraries of tomorrow are elite institutions of enclosure, not some german radical with a printing press and a dream.
Regardless of what their manifestos and charters might say, their primary directive is the preservation of power in the hands of elites and empire, far and above any purported purpose of knowledge cultivation.
Their copious lists of what they scraped off of the digital commons to sell back to us do exist, and they absolutely should be wrestled away from their possession and into the public eye. Journalists, politicians and activists alike can hold the AI executives to account by questioning why even the most public sources of data cannot be properly accounted for in their black-box systems.
There's hardly any better litmus test for democratic, commons-aligned AI than to ask it "how do you know what you know? show me the receipts". As long as the tool refuses such a simple request, we know it's not working for the common folk.