I'm just starting to dip my toes into recognizing gestures on a touchscreen, which feels like a kiddie pool for your project.

CC Marcel Goethals Ivan Reese

FWIW: I’ve been playing with parsing non-textual language. Conclusion: grade school math plus relational languages (PROLOG in my case) plus only a small handful of relations goes a long way towards something usable:

• /recognizing/ symbols and gestures is orthogonal to /parsing/ symbol-based languages ; “editing” and “parsing” and “semantics checking” should be completely orthogonal (accidental complexity abounds when the above is conflated together)
• containment
• connections
• bigger/smaller
• colour/shape/etc
• SVG has just everything needed (in fact, I ignore a lot of SVG, no swoopy art, just boxes and arrows and ellipses)
• blocks of text are “symbols”, their contents can be written in any 3GL, then parsed as black boxes along with the above
• real humans think of rectangles as isolated, stand-alone thingies --> isolation is key (I call it “0D”)
• currently using draw.io to edit drawings, then manually-written code to parse (e.g. hand-written semantics code using XML parser which sucks symbols (and some relationships) out of diagrams)
• ignore stuff that is hard to parse (swoopy stuff can remain in the diagrams, but is treated like comments)
• I transpile the hybrid diagrams to code in some existing language(s), or, run them directly without transpilation
• my friends: Ohm-JS, PROLOG, PEG, backtracking parsing, (I think miniKanren would work, too)

Thinking about syntax based on proximity and containment rather than adjacency.

Could you expand on the difference and/or tension here?

Christopher Shank Yes definitely! Conventional text-based programming languages are generally built on adjacency. This is a visual property in a way but I suppose a discrete one -- two things (characters, or words) are either adjacent or not -- so we don't tend to think of it as visuospatial.

Box-and-wire dataflow languages like max/pure data are built on connectedness, another discrete property. We tend to think of these as more visuospatial because you can arrange things how you like in a 2d arrangement, but this is all secondary notation rather than syntactical or semantic.

Proximity is where the arrangement enters the core language - things connect if they are proximal. This means you can assign additional meaning from how proximal things are. A really nice, successful example of this is the reactable. I'm not exploring this in the particular demo at the top of the thread, but think there are a lot of possibilities here. I explored this sort of thing some years ago with a haskell-based FRP front-end.

Containment I guess is actually a separate issue. In this demo I'm exploring just drawing around groups of glyphs. In text-based systems we use parenthesis for this.

One nice thing about exploring proximity is the possibilities for collaboration. The reactable is again a really nice case study for this - they made it circular so there's no way 'up' and people can collaborate by standing around it. Way ahead of its time really.

Hmm, first try and it attributes my work to someone else as a 'key fact' 🤔

and the second try. my fault for searching up my own projects I guess..

Yeah, it’s LLM based — it is hinted at during generation, but I guess it should be much clearer that’s the case and that it might be prone to some hallucination — sorry about that 🙏

Thanks for trying it out though!

I struggled to make it give a list of key facts that didn't contain lies, and of course they're all unsourced. Do you think LLMs have a place in research?

As it is, it seems dangerous to me.

With this experiment we’re trying to learn if this kind of usage pattern i.e. generating interactive initial content for a spatial workspace would be an interesting thing to explore further and what are the trade-offs.

The quality of generated results could be improved even now, as e.g. the models we are using at the moment are not the best possible.

Before we determine, if it’s worth doubling down on this effort we’re trading of that quality for price and speed.

This generative aspect is mostly for getting you started on exploring some topic, there are other little tools and features in the app that should take you bit further.

There might even an argument made that an inclusion of some more sophisticated verification tool for the generated content would be beneficial—that is the kind of thing I’m interested to learn, so thank you!

I think the LLMs most likely are going to be widely used for research whether we like it or not. Given that I think it’s worth exploring the tools and interactions around them to find what’s useful, humane and fun 🙂

Yes sure LLMs could be useful for researchers, but I think labeling statistical jumbles as key facts isn't humane, and in the world of research will result in embarrassing retractions and probably legal issues. Sorry to be so down on this, I'm sure there are nice aspects to this work too, but I don't find it a nice experience to be presented with falsehoods like this.