Hi all, I wanted to share an article I recently published in ACM Interactions: Reclaiming the Computer through LLM-Mediated Computing (it ended up being the cover story).
The piece is a mix of critique and speculation. I argue that today’s application-centered computing reduces the computer to icons and menus. I then suggest that with LLMs, we may be at the cusp of a shift toward a more relational and open-ended form of computing, where functionality emerges in dialogue rather than being predefined. I try and portrait a view of how that may look like and what it means.
I’d be really curious how this community reads it:
• Does it resonate with the kinds of directions you’ve been exploring here?
• Are there prior projects or framings I should know about that connect (or clash) with this line of thought?I’ve been listening to the podcast but just recently joined Slack. My sense is that many of you have been wrestling with similar questions. I’d love to hear your thoughts and how this fits (or doesn’t fit) with your own visions of computing’s future.
In a sense I think the way I think about this is a way to get to Alan Kay's idea about the computer as a medium. Not as a medium for text, but as a computational medium. His initial idea really was that computers should not come with pre-packaged applications, but the building blocks with which you can think and express. That's how I envision LLM-mediated computing could be. A modality of computing where through ongoing interaction, the computer and what we can do with it emerges in a sort of dialogue.
Ever since the desktop metaphor came about at Xerox PARC in the 1970s, we have interacted with computers that present themselves and their capabilities with what they display on-screen....
Your putting relational as the first description of this new approach is very much in tune with my thinking.
The tasks-not-tools paradigm is I think naturally relational, and while an LLM-driven interface such as you describe is a wonderful, natural direction to go, I think you can separate that from the underlying organisational and UI paradigm. When you do, you get my project, which is rebuilding computing around a (not-SQL!) relational paradigm.
FREST is simpler computing for ordinary folks and programmers alike.
Thanks for sharing, it's very related to this community imo! Here are some of my thoughts:
- I think the most compelling future of computing starts with the abolishment of apps because of the way they symbolize the commodity production of software which is largely the only branch of computing that we've experienced (I tend to call this particular branch techno-optimistic capitalism ). This also includes imagining computing outside of the office desktop GUI that every single person in the world interacts with. Let's call this alternative post-appitalism .
- We need alternative visions that situate themselves in existing context of use. Meaning we need ways to compost and recycle apps and the rest of the stack. I've done some writing that lays out some potential approaches to do this. While I hope that live/visual/malleable/end-user programming offer a vision of what can be they similar struggle to situate themself in existing context's of use. It's quite similar to what you point out with LLM research.
- I think the criticism of intention and context-switching is good, but I think the criticism that the application-paradigm of software fails to meet the "long tail of user needs" (figure 1) much more compelling and how it relates to (individual and collective) agency, self-governance (the web is implicitly feudal), and ownership of data and computation. All of these things we've given up large amounts of in the app-based paradigm. Even more pressing is the power dynamics being exerted around the techno-fascism of big tech, post=appitalism is about freeing ourselves from the oppression and dependence on the software produced by these private companies.
- AI/LLM could be part of the solution and I don't intend to be dismissive of it), but I continue to worry about their relationship to computational literacy. Literacy in computing has remained a square wave (figure 2). The GUI/desktop raised the floor of literacy, but the wall to being able to use a computer to it's "full potential" still requires much effort and education. In a similar way, I think LLMs (when used for vibe coding) might raise the floor of literacy, but wont smooth out the curve. This is because computing continues to remain opaque in the desktop GUI and LLMs, they provide little ways to understand what the produced software is actually doing. We need to figure out how to smooth out this curve (figure 3).
- It's worth noting that even the earliest prototypes at Xerox Star had different conceptions of documents and applications (figure 4). Documents were this heterogenous concept that could embed all types of interfaces and applications allowed one to author certain interfaces.
- Perhaps most pressing is the legal side of the app-paradigm around digital IP/copyright. Before those laws were a thing, legal scholars predicted the network effects/aggregation, incompatible apps/products, and heterogenous needs would be systematically underserved (figure 5/6). The DMCA made it illegal to tinker with the software of any app. LLMs alone cant fix this.
Great conversation starter! I think the call for 'slow technology' is new to me, but certainly resonates with Stengers' slow science and Schön's reflective conversation. As Christopher sketches out that spectrum of computational literacy, I can see the LLM as one strategy for hiking up the steep parts, but certainly not the only strategy. Knowledge of the 'toolbox' is not just knowing how to put pen to paper, it is also understanding what type of mark to make; and reasoning through the diagram, model, or inscription.
[...] the LLM generates the instructions that the computer executes. It serves as an interpretive layer that translates human expressions of intention into machine-level instructions. [...] What this requires is a runtime environment where those generated instructions can be executed reliably. While the LLM can compose the instructions, such an execution layer is still largely missing in contemporary computing.
Overall I think this insight is spot on, but I'm not sure why the generated instructions are not executed reliably? Isn't that the point of query languages, grammars, DSLs, and APIs, all ways of composing what Guyren is describing as functional programming operations?
An important point that I don’t often see discussed is that until AI becomes not just smarter than its user but considerably more reliable, we need a way to check its work. So the form in which it expresses that work should be easy to check.
Along the same lines, a better representation is a very powerful think (Playfair and all that). If I’ve a need to understand what my digital buddy is doing and why, we need the best possible expression of that stuff.
All of which is to say: I don’t think we’ll ever want to just always communicate with our computers in speech as they do in Star Trek. We’ll still need spreadsheets, databases (but please not with SQL) and so on. We can make them better, more facile. We can invent new ways of presenting information.
This will be enabled and accelerated by AI, enhanced by AI. But it is a UI and communication design issue, fundamentally.
📝 How the CIA Ruined Programming
We’re only just now starting to recover. And we have a long way to go.
Chiming in along the same lines as others here: LLMs probably have their place in a relational approach to computing, but they are not strictly necessary to get there.
Most of my daily work is done in two software universes that already operate relationally: Emacs and Glamorous Toolkit. Both of them make functionality accessible and discoverable through a search interface. I type one or two keywords related to what I want, and I get a list of choices to pick from.
The main progress I'd expect from LLMs is formulating more complex queries, and better handling of synonyms. But that's incremental rather than revolutionary, and as @Guyren Howe pointed out, it must be weighed against the errors introduced by LLMs.
Glamorous Toolkit is the Moldable Development Environment.
@Guyren Howe FREST is really intriguing: are you working to implement it or is it more a thought exercise? Have considered event sourcing/cqrs as an alternative to crdt?
@Jari I am playing the long game. I’ve started on a FileMaker like app that will integrate easily with services and data sources as well as local tables. This will form the funding source for the project.
To get there, I have to support myself while I work on it. So I’m mostly working on a less ambitious app to sell that will support me in the interim. And life keeps getting in the way of that.
At some point, I hope to interest a philanthropist or angel investor who would let me put together a team.
Awesome to see an engaging conversation!
Konrad Hinsen thanks for the link to glamorour toolkit. Looks interesting! I'm not saying LLMs are needed for relational computing. There has been many suggestions for this in the past, and many critiques of the wimp paradigm of course. But the notion of LLM-mediated computing I argue also gives rise to a relational perspective on computing, in contrast to wimp.
@Guyren Howe definitely agree that representations are useful. In fact we need them! But we should be careful with what representations we pick, what they end up representing, and how it shapes our world.
@Jasmine Otto what I mean by reliable is meant to foreclose any critique about running LLM generated code in an unsafe environment. Maybe reliable is the wrong term here, but the point is I think we need a robust substrate in which the code is run so as to deal with unforeseen issues, backtrack, and repair.
Christopher Shank thanks for these in depth comments. Haven't really thought of the long tail of apps argument before, and I definitely agree!
There is also a good essay on this topic:
Bringing AI coding tools into today’s software ecosystem is like bringing a talented sous chef to a food court. If you’re used to purchasing meals from a menu, a skilled chef can’t do much to help you. Similarly, if you’re using closed-source software from an app store, an AI coding assistant can’t do very much to help you as a user. To fully take advantage of the capabilities of AI, we need to move past the food court to something more like a kitchen—a site of open-ended creation.
📝 Malleable software: Restoring user agency in a world of locked-down apps
The original promise of personal computing was a new kind of clay. Instead, we got appliances: built far away, sealed, unchangeable. In this essay, we envision malleable software: tools that users can reshape with minimal friction to suit their unique needs.