You are viewing archived messages.
Go here to search the history.

Jim Meyer 🕰️ 2024-02-11 07:19:52

New visual programming languages don't fail as visual programming languages.

They fail as programming languages, which is what most new programming languages do.

Kongwei Ying 2024-03-15 11:59:21

The best way I would describe it is like this. Let's say that reality is like a state computation machine. How would it be possible to create a more accurate model of reality than we have currently? It would have to go beyond words, and would have to include visual, kinaesthetic constructs. It would have to be a higher dimensional representation, to the extent that we as humans can perceive a higher dimensional representation. So this means, higher than the 1D text we have, or its 2D UI, single frame in time representation. This would look like some kind of 4D temporal model, where it's possible to see the execution of a program in its entirety, all at once.

It would not only show semantic relationships in a graph sense, but also model the data structures and their operation over time. So the idea of "stocks" of data, and "flows" of data (from Donella Meadows, Systems Thinking) would apply here. React.useState might be represented as a small orb of energy, and if we setState, then it would visually change. When we assign const a = b, then it would visually indicate a flow of information from b -> a. I think this would be a much clearer representation of what a program is doing, and free up cognitive load to focus on more important tasks, as opposed to the programmer having to hold a mental model in their head. No matter how good a programmer is, if he or she possesses a better more accurate kinesthetic model of what is happening, their brain power is freed up to solve more problematic and relevant abstractions.

I've been mocking up a prototype of how this would work, and the basic idea is overlaying a tool like VSCode with "vibes" as well as flow relations. It's like an annotation layer over existing code, but this is just the basic start of how it would look like. Ultimately, it would look like a mini universe. for a non-programming context demonstration of what I'm talking about.

Dave Liepmann 2024-03-12 09:49:47

Jakob Nielsen asserts that accessibility has failed to serve disabled users because of a mistaken focus on standards. This approach is unable to address the wide variety of disabilities, and because it is by definition a derivative of a GUI, it is doomed to provide a second-class experience to e.g. blind users. He points out differing needs by older and less literate users, and proposes generative UI as a promising solution.

I was skeptical of the headline and tweet-length summaries. There has been not-insignificant backlash. The full article leaves me still skeptical but certainly willing to see where the idea of bespoke-to-one's-needs UIs could take us.

Duncan Cragg 2024-03-12 10:27:33

Wow, this reminds me of work I did in 1989 on "intelligent user interfaces" for a company called EASAMS, my first graduate job.

It didn't get anywhere because (a) neural networks and AI generally weren't up to it and (b) the in-house sponsors didn't really think it was worth further time investment.

I'm struggling to remember the details, but it was about inferring the user's intentions and goals, then adjusting the UI in real time to support them. So the UI became an extension of the user.

In its simplest incarnation, to get over those moments when you shout at the computer "but you already knew that!!!"

Duncan Cragg 2024-03-12 10:29:02

(I didn't and wouldn't work for the military ever again, btw)

Eli Mellen 2024-03-12 12:09:23

If you are going to read Nielsen’s piece, I encourage you to also check out this response from Brian DeConinck

Brian is, in my opinion and experience the most down-to-earth and expert accessibility voice actually doing the work everyday.

Lu Wilson 2024-03-12 12:37:59

it always reminds me of the ramp VS bionic legs thing.

let's say your venue has some stairs at the front door, and people in wheelchairs struggle to enter it, there are various things you can do:

  • carry the person up the stairs.
  • add a ramp (in addition to the stairs).
  • replace the ramp with some stairs (and never build stairs again).

This is a common example that gets taught in teacher training for how to think about accessibility in the classroom (context: i used to work in Special Educational Needs).

the best option in this metaphor is to start by building a ramp, then everyone can enter together, and you don't have to do any extra work, and everyone can be independent.

I've been using this metaphor recently in tech accessibility, because the proposed solution is often something like:

  • build an AI assisted Iron Man megasuit that lets the person fly

in this case, thats what it seems like to me. I guess there's a common mentality that technology will be the only __ answer to all our problems. I quote: "Supporting Disabled Users Requires Generative UI".

i think that we need ramps, not iron man suits, which requires societal change, not technological change

Dave Liepmann 2024-03-12 12:46:14

Re: DeConinck's interpretation of Nielsen as saying "2. Some kind of automated tool detects information about the user, including their disability status." — the idea I had was more that the disability status would happen user-side, or as an interaction

Dave Liepmann 2024-03-12 12:50:09

also, thanks for linking — the other critiques I'd seen were far more thin

William Taysom 2024-03-12 13:13:25

Consider the intrinsic potential malleability of software compared to the rigidity of the built environment. Perhaps the software equivalent to the ramp is the ability to easily perform one-off customizations of things. Hmm, concrete real examples...

William Taysom 2024-03-12 13:23:55

Maybe this is my Iron Man suit Mark 0 since all of these are hacks around general UI limitations:

  • I want to load the audio of some page into my podcast player. In instances that are relevant to me, I can from my browser type Command+Shift+C, select an element near the play button, look for an <audio> tag, copy a URL, and then past into Overcast's Uploads form.
  • I want to extract text from an image. I press Command+Shift+0 and Screenotate tries its best.
  • I need to use a website in a language I mostly don't know. I select text, and then click on the Google Translate button that appears whenever I do this... except when I can't select the text, then I fall back to a combination of (1) and (2).
  • I just wanted to do this right now, but don't have the tooling for it. But if I did, I could say to my LLM, "On this page, select the year 2022, download the documents for the four quarters that appear. Then repeat for 2023." — If any of you have a tool that does that, please, please let me know.

I guess the common thread is that I want to be able to choose what I do with software without its participation or consent.

Andrew F 2024-03-12 15:54:02

I don't know if the ramp vs stairs analogy is useful here. In software, what's a ramp to some people is a wall to others. An audio interface that helps visually impaired people does nothing for the deaf. If you try to accommodate them both with touch, you'll lose quadriplegics. There's no intersection of interaction capability.

Andrew F 2024-03-12 16:13:18

One of the problems with this approach is the inconsistency of the UI. People always need to learn the UI a little bit. Unless the AI is perfectly deterministic, which it won't be, it'll make little changes to the UI, maybe even time. This is an even worse version of the constant human-driven UI changes that users already hate with burning passion and contempt.

Related: the UI will be impossible to document, and almost impossible to help anyone with. People can't necessarily share their knowledge about it between them. Even official support will probably involve screensharing at best; I have no idea how that works for people with disabilities. Good luck writing support scripts (actually we can take that one as a silver lining, but companies won't like it).

Dave Liepmann 2024-03-12 16:52:12

Re: the ramp analogy, it doesn't take into account Nielsen's point that critiques accessibility standards for being a derivative of the non-accessible design. I acknowledge DeConinck's counterargument that "Before you design your interface, you could make a plain text outline", which would make the accessibility approach...somewhat less of a derivative? Still. What proportion of the web is built that way? When I estimate the proportion I think we could achieve, the analogy that comes to mind is more elevators than ramps. I'm pro-elevator but we can't just hand-wave away the effort of redesigning our built environment and ways of working.

Andrew F 2024-03-12 16:55:03

What proportion of the web is built with generative UI? And why does that matter? Clearly most of the web is bad for usability, and if you care about that then you're already resigned to making big changes.

Dave Liepmann 2024-03-12 16:59:42


In other words, some users get the full experience, the one with all the words, all the context, and all the options. But if Nielsen’s AI thinks you have a disability, you’ll get a different experience, a simpler experience that’s more appropriate for people like you . It’s an ugly kind of paternalism with a new AI twist.

In contexts where I have been disabled, yes — this is precisely what I wanted. Less context, less precise, more gist, simpler, more appropriate for someone like me.

Where I diverge from his vision of how this has to work is that I want control over the algorithm to be in my hands. I say what my needs are today, with all knobs available to be tuned.

Since it’s just too hard to follow accessibility best practices—plain language, a coherent document structure, support for a broad set of user agents—let’s just design UIs for us regular people and let the algorithm dumb it down for the edge cases.

You literally cannot persuade the German people to write in plain language with a coherent document structure. It's not a realistic proposal; yes it is too hard. I would prefer a harm reduction strategy that works around the reality that UIs are designed for "regular people" [sic] and alternatives must be based on that fundament.

Dave Liepmann 2024-03-12 17:01:50

What proportion of the web is built with generative UI?

Sorry — I meant, what proportion of the web is built with DeConinck's accessibility-first design approach? I'd even accept the lesser standard of accessibility to the level of, say, the UK government's requirements.

Andrew F 2024-03-12 17:02:31

I know what you meant. My point is that it's a bad question.

Dave Liepmann 2024-03-12 17:02:40

Sorry, I hear you now

Dave Liepmann 2024-03-12 17:04:17

If the tools for generating UIs on the fly is in my hands the way translation tools or screen readers are in my hands, I'm not sure your point holds

Dave Liepmann 2024-03-12 17:05:00

(preferably more than those tools, sure, but still)

Andrew F 2024-03-12 17:10:55

In your hands as the end user? I don't think that's what Nielsen is talking about. I don't think it's all that likely, either. Most companies aren't interested in offering the full featured APIs to their customers that would be needed to make that work. They could do that today if they wanted, but they don't.

If you mean in your hands as a developer, that's not even a hypothetical for accessible HTML or outline-oriented UI, you can do that right now. So I don't see how that puts generative UI ahead, either.

Konrad Hinsen 2024-03-12 17:11:07

The ramp example (however suitable it may be as an analogy for software) actually illustrates the absence of a universal solution for accessibility problems. There are people who can use stairs but not ramps, because their ankles are too stiff to deal with the slope.

Eli Mellen 2024-03-12 17:19:18

An angle perhaps missing from consideration here, but gestured at, is assistive technology, like screen zoom, screen readers, braille displays, voice control, etc.

They offer alternative modes of mediation between person and interface.

Lu Wilson 2024-03-12 19:38:45

I'm still team ramp TBH. ie: i believe we've had the technology for accessibility for a long time. a new and even more complicated and even more expensive technology is not going to be the thing that makes accessibility better. if anything, i think the natural inclination (as we see here) is to use it as an excuse to care less about accessibility. am i just being cynical?

Andrew F 2024-03-12 19:51:23

Ramps are a honking great idea, let's do more of those. I just don't think they're a total solution. I don't think there can be one.

Christopher Shank 2024-03-12 21:19:31

Related to building ramps, the 1st edition of "Teaching Accessible Computing" was just released and is free to the public!

For computing to work for everyone, it must be accessible to everyone. Alas, it is not: people with disabilities in mobility, vision, hearing, learning, attention, and more regularly face software that is hard or impossible for them to use. One reason for this is that when we educate future software engineers, we rarely teach them anything about accessibility. This limits their ability to find and fix accessibility defects and advocate to their organization to prioritize those fixes. More importantly, it limits the capacity of software organizations to design software that is accessible from day one.

Eli Mellen 2024-03-12 21:48:55

My sort of trite way of framing it:

Whereas mobile-first design and development invited folks to think more expansively about the physical reality of the devices people use, ~accessibility-first design and development invites folks to think more expansively about the lived experiences, and physical reality of actual people.~

Konrad Hinsen 2024-03-13 07:51:10

Lu Wilson I agree that many accessibility issues can be solved with old and well-known tech. I think it doesn‘t happen because people don‘t look at the tech from that perspective. And for the hard-core tech people, accessibility is not a priority because there is no glory to be found there.

Joe Nash 2024-03-13 07:53:09

When Nielsen talks about the UI being completely different for every user, and even on every use of the app, he only stops to soothe the designer. This is wildly off-track to me: the ability to become familiar with and make the app predictable is really important to several classes of disabled users. Or for that matter, all users! Who wants a UI that changes on them all the time? I’m also very intrigued how he envisages support working, both at the local scale by those surrounding disabled users, and “at scale” by app builders serving their customers

Lu Wilson 2024-03-13 12:35:33

guys it's simple we need to make accessibility sexy

Eli Mellen 2024-03-13 16:12:50

Ironically, Nielsen got slapped once upon a time for exactly that

Eli Mellen 2024-03-13 16:13:26

if you look up the Nielsen groups training videos on accessible design they’re almost all delivered by tall, white ladies

Christopher Shank 2024-03-12 23:17:12


You’ve heard of C, C++, C#, objective-C… well, Cmajor is a C-family language designed specifically for writing DSP signal processing code.

They have a JIT engine, but it also compiles to WASM so it can run on the web. This means old synths can be ported to run on the web!

Oleksandr Kryvonos 2024-03-13 19:22:40

this is a great inspirational talk about sharing ideas

Jeffrey Tao 2024-03-13 20:11:43

Haven't checked in on Dark in a few years but it seems they're dropped the custom editing environment, among other changes

📝 An overdue status update on Darklang

We've been working hard at Darklang for the past year, but haven't been very vocal about what we've been up to.

Here’s the “Darklang” that’s been live for years:

Darklang – the live version, which we're now calling Darklang classic – is a developer tool composed of a few interconnected

Ivan Reese 2024-03-14 04:59:17

A call for talks for the first ever workshop on Choreographic Programming at PLDI this summer. Deadline next Friday.

📝 CP 2024 - Choreographic Programming 2024 - PLDI 2024

Choreographies are coordination plans for concurrent and distributed systems. A choreography defines the roles of the involved participants and how they are supposed to work together. In the emerging paradigm of choreographic programming (CP), choreographies are programs that can be compiled to executable implementations. CP originated primarily in the context of process calculi, with preliminary work done to establish its foundations and experiment with implementations. Recently, several proposals have shown that one can adapt CP to work in synergy with mainstream programming paradigms ...

Ionuț G. Stan 2024-03-14 08:15:40

Alternative link (that works for me):

For a moment I thought it was about dance notation, like Labanotation 😅

📝 CP 2024 - Choreographic Programming 2024 - PLDI 2024

Choreographies are coordination plans for concurrent and distributed systems. A choreography defines the roles of the involved participants and how they are supposed to work together. In the emerging paradigm of choreographic programming (CP), choreographies are programs that can be compiled to executable implementations. CP originated primarily in the context of process calculi, with preliminary work done to establish its foundations and experiment with implementations. Recently, several proposals have shown that one can adapt CP to work in synergy with mainstream programming paradigms ...

📝 Labanotation

Labanotation (grammatically correct form "Labannotation" or "Laban notation" is uncommon) is a system for analyzing and recording human movement (notation system), invented by Austro-Hungarian choreographer and dancer Rudolf von Laban (1879-1958, a central figure in European modern dance), who developed his notation on movements in the 1920s.

Ivan Reese 2024-03-14 14:02:04

Thanks — fixed the link in my post too.

Alex McLean 2024-03-14 14:15:05

Yes I was excited to see a choreographic workshop at PLDI

Alex McLean 2024-03-14 14:16:08

(Laban was a bit of a Nazi choreographer)

Alex McLean 2024-03-14 14:17:30

If anyone's interested in the other kind of choreographic programming, here's a journal article I recently wrote with Kate Sicchio

📝 Live Notation for Patterns of Movement | TDR | Cambridge Core

Live Notation for Patterns of Movement - Volume 68 Issue 1

Alex McLean 2024-03-14 14:17:30

If anyone's interested in the other kind of choreographic programming, here's a journal article I recently wrote with Kate Sicchio

📝 Live Notation for Patterns of Movement | TDR | Cambridge Core

Live Notation for Patterns of Movement - Volume 68 Issue 1

Ionuț G. Stan 2024-03-14 14:27:51

(Laban was a bit of a Nazi choreographer)

Following a rehearsal of choreography he had prepared for the 1936 Summer Olympics in Berlin, Laban was targeted by the Nazi party. He eventually found refuge in England in 1937.

Alex McLean 2024-03-14 14:29:58

Yes he choreographed for Hitler but the regime rejected him (rather than vice versa)

Alex McLean 2024-03-14 14:32:18

Not to dismiss his work, just good to acknowledge these things. From my paper with Kate:

"Reviewing related historical and contemporary approaches to movement notation allowed us to reflect upon and bring together the different histories of code and choreography. A key reference point in the representation and notation of human exertion as effort can be found in Effort: Economy in Body Movement written by choreographer Rudolf Laban in collaboration with management consultant F.C. Lawrence in 1947. The book proposes a systematic, seemingly elegant way to categorize human efforts in movement, through a combination of elements such as time, weight, and space. Effort is as applicable to industrial time-and-motion studies as it is to dance choreography. We should also note that Laban’s earlier work was as choreographer for Hitler and Goebbels, until his work on the 1936 Olympic games was rejected (the Nazis rejecting Laban, rather than vice versa). While this theory arose from white supremacy and Fordism, it continues to be influential in choreographic research and practice."

Alex McLean 2024-03-14 14:33:43

I do enjoy his work on efforts etc though. I just think it's a shame when references begin and end with a single problematic white male so often.. Ignoring much longer and richer histories

Alex McLean 2024-03-14 14:34:02

(sorry for the topic drift)

Don Abrams 2024-03-18 00:21:52

this has been my major motivation recently

There seems to be a bit split between top-down, where you describe a global choreography (generate a program per actor) vs generating global properties (c hecking guarantees by combining the behavior of different actors)

Alex McLean 2024-03-14 14:36:43

I've only just noticed that there is an arts track at the ongoing <programming> conference this year! awesome

📝 ‹Programming› 2024 - Arts - ‹Programming› 2024

Step into a world where lines of code dance with strokes of creativity! At this year’s edition of the Programming conference, we are opening an Arts track to engage with the idea that art can inform computational practices and their presence in the world. With this track we invite participants of the conference to join a series of presentations and meet with the artists. We hope the Arts track will inspire the programming community to propose artistic projects for future editions. Events include:

KhipuKoding, a session with live coding by Paola Torres Nunez del Prado; Programming a ...

Lu Wilson 2024-03-15 10:21:10

Update from Dave Ackley (whose essay we talked about on the last episode)

A large part of this video is about some initial work on making the 'demon horde sort' better, which is his proposed robust-first way of doing number sorting

Kartik Agaram 2024-03-15 13:37:55

Ah, subsumption architecture takes me back to this little project of mine:

Christopher Shank 2024-03-16 07:03:21

“If What We Made Were Real:

Against Imperialism and Cartesianism in Computer Science, and for a discipline that creates real artifacts for real communities, following the faculties of real cognition” by Basman (2020)

Christopher Shank 2024-03-16 16:50:52

Here are a number of kinds of power, widely considered traditional powers amongst software engineers, that we should give up:

  • The power to create grammars with infinite numbers of valid sentences
  • The power to construct programs that might consume unbounded time and/or space, or perhaps never terminate
  • The power to hide pieces of state behind abstractions (APIs or other kinds of interfaces)
  • The power to construct pieces of software through irreversible or nearly irreversible machines such as compilers
  • The power to divide up a particular domain into a single hierarchical decomposition of entities with properties, connected by relations
  • The power to prescribe the exact sequence of operations needed to achieve a particular result
  • The power to import machinery, definitions or methodologies from related disciplines, without evaluating their tendency to result in appropriate products
  • The power to change the form or behaviour of a program in an updated version, without giving a cost-free (to both users and developers) choice to retain the old form
Christopher Shank 2024-03-16 17:00:51

All of the "fake software" we have today is not like this, and does not allow this. Instead, it consists of a number of "locks" through which water only flows in one direction - conducting power out from the worlds of developers into the worlds of users, and accepting no inflow in the other direction. This is precisely what APIs and abstraction boundaries, compilers and modules are designed to achieve - to concentrate power in the hands of those who have it, and to ensure that none of it leaks outwards.

Duncan Cragg 2024-03-16 20:32:09

Wow this is just my kinda stuff, I'll read that, then hopefully come by here again with my usual Deep Insights...

Christopher Shank 2024-03-16 23:24:47

Cant wait to hear your thoughts!

Duncan Cragg 2024-03-16 23:41:31

I've just started reading it now - and I'm nodding along furiously. I'll have to finish reading it tomorrow as I've had a glass of wine so won't remember it well enough! Anyway, here's one of my articles bemoaning the same issue (in a more trivial way perhaps), and offering my solution - which is "The Inversion":

📝 Turning the Blob inside out

Idiosyncratic apps come between you and your data...

Ivan Reese 2024-03-17 00:30:00

(Possible spoiler: we're considering doing an episode of the podcast on this paper; I'm going to save my thoughts for that)

Justin Janes 2024-03-17 03:01:16

This really feels like there is a pendulum that has almost never swung away from one extreme for computation for decades and we’re close to realizing it.

Duncan Cragg 2024-03-17 10:48:10

OK, I've read the doc, and I agree with all the sentiments and motivations and broad concepts, but of course, don't agree with the details of the solution side! Basman's solution alludes to Declarative programming as a better alternative to Imperative, but doesn't call for that jump as explicitly as I do, or recognise it as a choice between duals: what I call the "Inversion". Here's another one from my blog on that:

📝 The Inversion

Techie article! The Inversion is my architectural vision: freeing our data into a virtual world.

Duncan Cragg 2024-03-17 15:30:29

We shoulda pinged the author! @Antranig Basman 🤗