The best way I would describe it is like this. Let's say that reality is like a state computation machine. How would it be possible to create a more accurate model of reality than we have currently? It would have to go beyond words, and would have to include visual, kinaesthetic constructs. It would have to be a higher dimensional representation, to the extent that we as humans can perceive a higher dimensional representation. So this means, higher than the 1D text we have, or its 2D UI, single frame in time representation. This would look like some kind of 4D temporal model, where it's possible to see the execution of a program in its entirety, all at once.

It would not only show semantic relationships in a graph sense, but also model the data structures and their operation over time. So the idea of "stocks" of data, and "flows" of data (from Donella Meadows, Systems Thinking) would apply here. React.useState might be represented as a small orb of energy, and if we setState, then it would visually change. When we assign const a = b, then it would visually indicate a flow of information from b -> a. I think this would be a much clearer representation of what a program is doing, and free up cognitive load to focus on more important tasks, as opposed to the programmer having to hold a mental model in their head. No matter how good a programmer is, if he or she possesses a better more accurate kinesthetic model of what is happening, their brain power is freed up to solve more problematic and relevant abstractions.

I've been mocking up a prototype of how this would work, and the basic idea is overlaying a tool like VSCode with "vibes" as well as flow relations. It's like an annotation layer over existing code, but this is just the basic start of how it would look like. Ultimately, it would look like a mini universe.

www.eusaybia.com for a non-programming context demonstration of what I'm talking about.

Wow, this reminds me of work I did in 1989 on "intelligent user interfaces" for a company called EASAMS, my first graduate job.

It didn't get anywhere because (a) neural networks and AI generally weren't up to it and (b) the in-house sponsors didn't really think it was worth further time investment.

I'm struggling to remember the details, but it was about inferring the user's intentions and goals, then adjusting the UI in real time to support them. So the UI became an extension of the user.

In its simplest incarnation, to get over those moments when you shout at the computer "but you already knew that!!!"

(I didn't and wouldn't work for the military ever again, btw)

If you are going to read Nielsen’s piece, I encourage you to also check out this response from Brian DeConinck

Brian is, in my opinion and experience the most down-to-earth and expert accessibility voice actually doing the work everyday.

it always reminds me of the ramp VS bionic legs thing.

let's say your venue has some stairs at the front door, and people in wheelchairs struggle to enter it, there are various things you can do:

• carry the person up the stairs.
• add a ramp (in addition to the stairs).
• replace the ramp with some stairs (and never build stairs again).

This is a common example that gets taught in teacher training for how to think about accessibility in the classroom (context: i used to work in Special Educational Needs).

the best option in this metaphor is to start by building a ramp, then everyone can enter together, and you don't have to do any extra work, and everyone can be independent.

I've been using this metaphor recently in tech accessibility, because the proposed solution is often something like:

• build an AI assisted Iron Man megasuit that lets the person fly

in this case, thats what it seems like to me. I guess there's a common mentality that technology will be the only __ answer to all our problems. I quote: "Supporting Disabled Users Requires Generative UI".

i think that we need ramps, not iron man suits, which requires societal change, not technological change

Re: DeConinck's interpretation of Nielsen as saying "2. Some kind of automated tool detects information about the user, including their disability status." — the idea I had was more that the disability status would happen user-side, or as an interaction

also, thanks for linking — the other critiques I'd seen were far more thin

Consider the intrinsic potential malleability of software compared to the rigidity of the built environment. Perhaps the software equivalent to the ramp is the ability to easily perform one-off customizations of things. Hmm, concrete real examples...

Maybe this is my Iron Man suit Mark 0 since all of these are hacks around general UI limitations:

• I want to load the audio of some page into my podcast player. In instances that are relevant to me, I can from my browser type Command+Shift+C, select an element near the play button, look for an <audio> tag, copy a URL, and then past into Overcast's Uploads form.
• I want to extract text from an image. I press Command+Shift+0 and Screenotate tries its best.
• I need to use a website in a language I mostly don't know. I select text, and then click on the Google Translate button that appears whenever I do this... except when I can't select the text, then I fall back to a combination of (1) and (2).
• I just wanted to do this right now, but don't have the tooling for it. But if I did, I could say to my LLM, "On this page, select the year 2022, download the documents for the four quarters that appear. Then repeat for 2023." — If any of you have a tool that does that, please, please let me know.

I guess the common thread is that I want to be able to choose what I do with software without its participation or consent.

I don't know if the ramp vs stairs analogy is useful here. In software, what's a ramp to some people is a wall to others. An audio interface that helps visually impaired people does nothing for the deaf. If you try to accommodate them both with touch, you'll lose quadriplegics. There's no intersection of interaction capability.

One of the problems with this approach is the inconsistency of the UI. People always need to learn the UI a little bit. Unless the AI is perfectly deterministic, which it won't be, it'll make little changes to the UI, maybe even time. This is an even worse version of the constant human-driven UI changes that users already hate with burning passion and contempt.

Related: the UI will be impossible to document, and almost impossible to help anyone with. People can't necessarily share their knowledge about it between them. Even official support will probably involve screensharing at best; I have no idea how that works for people with disabilities. Good luck writing support scripts (actually we can take that one as a silver lining, but companies won't like it).

Re: the ramp analogy, it doesn't take into account Nielsen's point that critiques accessibility standards for being a derivative of the non-accessible design. I acknowledge DeConinck's counterargument that "Before you design your interface, you could make a plain text outline", which would make the accessibility approach...somewhat less of a derivative? Still. What proportion of the web is built that way? When I estimate the proportion I think we could achieve, the analogy that comes to mind is more elevators than ramps. I'm pro-elevator but we can't just hand-wave away the effort of redesigning our built environment and ways of working.

What proportion of the web is built with generative UI? And why does that matter? Clearly most of the web is bad for usability, and if you care about that then you're already resigned to making big changes.

DeConinck:

In other words, some users get the full experience, the one with all the words, all the context, and all the options. But if Nielsen’s AI thinks you have a disability, you’ll get a different experience, a simpler experience that’s more appropriate for people like you . It’s an ugly kind of paternalism with a new AI twist.

In contexts where I have been disabled, yes — this is precisely what I wanted. Less context, less precise, more gist, simpler, more appropriate for someone like me.

Where I diverge from his vision of how this has to work is that I want control over the algorithm to be in my hands. I say what my needs are today, with all knobs available to be tuned.

Since it’s just too hard to follow accessibility best practices—plain language, a coherent document structure, support for a broad set of user agents—let’s just design UIs for us regular people and let the algorithm dumb it down for the edge cases.

You literally cannot persuade the German people to write in plain language with a coherent document structure. It's not a realistic proposal; yes it is too hard. I would prefer a harm reduction strategy that works around the reality that UIs are designed for "regular people" [sic] and alternatives must be based on that fundament.

What proportion of the web is built with generative UI?

Sorry — I meant, what proportion of the web is built with DeConinck's accessibility-first design approach? I'd even accept the lesser standard of accessibility to the level of, say, the UK government's requirements.

I know what you meant. My point is that it's a bad question.

Sorry, I hear you now

If the tools for generating UIs on the fly is in my hands the way translation tools or screen readers are in my hands, I'm not sure your point holds

(preferably more than those tools, sure, but still)

In your hands as the end user? I don't think that's what Nielsen is talking about. I don't think it's all that likely, either. Most companies aren't interested in offering the full featured APIs to their customers that would be needed to make that work. They could do that today if they wanted, but they don't.

If you mean in your hands as a developer, that's not even a hypothetical for accessible HTML or outline-oriented UI, you can do that right now. So I don't see how that puts generative UI ahead, either.

The ramp example (however suitable it may be as an analogy for software) actually illustrates the absence of a universal solution for accessibility problems. There are people who can use stairs but not ramps, because their ankles are too stiff to deal with the slope.

An angle perhaps missing from consideration here, but gestured at, is assistive technology, like screen zoom, screen readers, braille displays, voice control, etc.

They offer alternative modes of mediation between person and interface.

I'm still team ramp TBH. ie: i believe we've had the technology for accessibility for a long time. a new and even more complicated and even more expensive technology is not going to be the thing that makes accessibility better. if anything, i think the natural inclination (as we see here) is to use it as an excuse to care less about accessibility. am i just being cynical?

Ramps are a honking great idea, let's do more of those. I just don't think they're a total solution. I don't think there can be one.

Related to building ramps, the 1st edition of "Teaching Accessible Computing" was just released and is free to the public!

For computing to work for everyone, it must be accessible to everyone. Alas, it is not: people with disabilities in mobility, vision, hearing, learning, attention, and more regularly face software that is hard or impossible for them to use. One reason for this is that when we educate future software engineers, we rarely teach them anything about accessibility. This limits their ability to find and fix accessibility defects and advocate to their organization to prioritize those fixes. More importantly, it limits the capacity of software organizations to design software that is accessible from day one.

bookish.press/tac

My sort of trite way of framing it:

Whereas mobile-first design and development invited folks to think more expansively about the physical reality of the devices people use, ~accessibility-first design and development invites folks to think more expansively about the lived experiences, and physical reality of actual people.~

Lu Wilson I agree that many accessibility issues can be solved with old and well-known tech. I think it doesn‘t happen because people don‘t look at the tech from that perspective. And for the hard-core tech people, accessibility is not a priority because there is no glory to be found there.

When Nielsen talks about the UI being completely different for every user, and even on every use of the app, he only stops to soothe the designer. This is wildly off-track to me: the ability to become familiar with and make the app predictable is really important to several classes of disabled users. Or for that matter, all users! Who wants a UI that changes on them all the time? I’m also very intrigued how he envisages support working, both at the local scale by those surrounding disabled users, and “at scale” by app builders serving their customers

guys it's simple we need to make accessibility sexy

Ironically, Nielsen got slapped once upon a time for exactly that

if you look up the Nielsen groups training videos on accessible design they’re almost all delivered by tall, white ladies

Alternative link (that works for me): pldi24.sigplan.org/home/cp-2024

For a moment I thought it was about dance notation, like Labanotation 😅 en.m.wikipedia.org/wiki/Labanotation

Thanks — fixed the link in my post too.

Yes I was excited to see a choreographic workshop at PLDI

(Laban was a bit of a Nazi choreographer)

If anyone's interested in the other kind of choreographic programming, here's a journal article I recently wrote with Kate Sicchio cambridge.org/core/journals/the-drama-review/article/live-notation-for-patterns-of-movement/CD9BC7EF97B132E90F71F078A3EA2F8E

If anyone's interested in the other kind of choreographic programming, here's a journal article I recently wrote with Kate Sicchio cambridge.org/core/journals/the-drama-review/article/live-notation-for-patterns-of-movement/CD9BC7EF97B132E90F71F078A3EA2F8E

(Laban was a bit of a Nazi choreographer)

en.m.wikipedia.org/wiki/Rudolf_von_Laban

Following a rehearsal of choreography he had prepared for the 1936 Summer Olympics in Berlin, Laban was targeted by the Nazi party. He eventually found refuge in England in 1937.

Yes he choreographed for Hitler but the regime rejected him (rather than vice versa)

Not to dismiss his work, just good to acknowledge these things. From my paper with Kate:

"Reviewing related historical and contemporary approaches to movement notation allowed us to reflect upon and bring together the different histories of code and choreography. A key reference point in the representation and notation of human exertion as effort can be found in Effort: Economy in Body Movement written by choreographer Rudolf Laban in collaboration with management consultant F.C. Lawrence in 1947. The book proposes a systematic, seemingly elegant way to categorize human efforts in movement, through a combination of elements such as time, weight, and space. Effort is as applicable to industrial time-and-motion studies as it is to dance choreography. We should also note that Laban’s earlier work was as choreographer for Hitler and Goebbels, until his work on the 1936 Olympic games was rejected (the Nazis rejecting Laban, rather than vice versa). While this theory arose from white supremacy and Fordism, it continues to be influential in choreographic research and practice."

I do enjoy his work on efforts etc though. I just think it's a shame when references begin and end with a single problematic white male so often.. Ignoring much longer and richer histories

(sorry for the topic drift)

this has been my major motivation recently

There seems to be a bit split between top-down, where you describe a global choreography (generate a program per actor) vs generating global properties (c hecking guarantees by combining the behavior of different actors)

Ah, subsumption architecture takes me back to this little project of mine: git.sr.ht/~akkartik/brooks-ruby-warrior

Here are a number of kinds of power, widely considered traditional powers amongst software engineers, that we should give up:

• The power to create grammars with infinite numbers of valid sentences
• The power to construct programs that might consume unbounded time and/or space, or perhaps never terminate
• The power to hide pieces of state behind abstractions (APIs or other kinds of interfaces)
• The power to construct pieces of software through irreversible or nearly irreversible machines such as compilers
• The power to divide up a particular domain into a single hierarchical decomposition of entities with properties, connected by relations
• The power to prescribe the exact sequence of operations needed to achieve a particular result
• The power to import machinery, definitions or methodologies from related disciplines, without evaluating their tendency to result in appropriate products
• The power to change the form or behaviour of a program in an updated version, without giving a cost-free (to both users and developers) choice to retain the old form

All of the "fake software" we have today is not like this, and does not allow this. Instead, it consists of a number of "locks" through which water only flows in one direction - conducting power out from the worlds of developers into the worlds of users, and accepting no inflow in the other direction. This is precisely what APIs and abstraction boundaries, compilers and modules are designed to achieve - to concentrate power in the hands of those who have it, and to ensure that none of it leaks outwards.

Wow this is just my kinda stuff, I'll read that, then hopefully come by here again with my usual Deep Insights...

Cant wait to hear your thoughts!

I've just started reading it now - and I'm nodding along furiously. I'll have to finish reading it tomorrow as I've had a glass of wine so won't remember it well enough! Anyway, here's one of my articles bemoaning the same issue (in a more trivial way perhaps), and offering my solution - which is "The Inversion":

duncancragg.substack.com/p/turning-the-blob-inside-out

(Possible spoiler: we're considering doing an episode of the podcast on this paper; I'm going to save my thoughts for that)

This really feels like there is a pendulum that has almost never swung away from one extreme for computation for decades and we’re close to realizing it.

OK, I've read the doc, and I agree with all the sentiments and motivations and broad concepts, but of course, don't agree with the details of the solution side! Basman's solution alludes to Declarative programming as a better alternative to Imperative, but doesn't call for that jump as explicitly as I do, or recognise it as a choice between duals: what I call the "Inversion". Here's another one from my blog on that: duncancragg.substack.com/p/the-inversion

We shoulda pinged the author! @Antranig Basman 🤗