You are viewing archived messages.
Go here to search the history.

Alex McLean 2023-10-06 10:29:27

Lately I've been experimenting with recognition of hand-drawn symbols in embedded structures, working with Luke Iannini in Realtalk

Thinking about how perceptually salient properties (such as spikiness/roundness, wonkiness) could be taken into account in a kind of analogue interpretation of the shapes alongside discrete symbol recognition as 'signposts' in feature space.

and what happens to those features when some symbols are marked out as higher order functions.

Thinking about syntax based on proximity and containment rather than adjacency.

also what happens when the parser itself is part of the scene.. e.g. how does its orientation change the parsing of the symbols?

Would love to hear about other projects that have explored this kind of area!

Kartik Agaram 2023-10-06 15:18:35

I'm just starting to dip my toes into recognizing gestures on a touchscreen, which feels like a kiddie pool for your project.

Paul Tarvydas 2023-10-06 16:22:21

FWIW: I’ve been playing with parsing non-textual language. Conclusion: grade school math plus relational languages (PROLOG in my case) plus only a small handful of relations goes a long way towards something usable:

  • /recognizing/ symbols and gestures is orthogonal to /parsing/ symbol-based languages ; “editing” and “parsing” and “semantics checking” should be completely orthogonal (accidental complexity abounds when the above is conflated together)
  • containment
  • connections
  • bigger/smaller
  • colour/shape/etc
  • SVG has just everything needed (in fact, I ignore a lot of SVG, no swoopy art, just boxes and arrows and ellipses)
  • blocks of text are “symbols”, their contents can be written in any 3GL, then parsed as black boxes along with the above
  • real humans think of rectangles as isolated, stand-alone thingies --> isolation is key (I call it “0D”)
  • currently using to edit drawings, then manually-written code to parse (e.g. hand-written semantics code using XML parser which sucks symbols (and some relationships) out of diagrams)
  • ignore stuff that is hard to parse (swoopy stuff can remain in the diagrams, but is treated like comments)
  • I transpile the hybrid diagrams to code in some existing language(s), or, run them directly without transpilation
  • my friends: Ohm-JS, PROLOG, PEG, backtracking parsing, (I think miniKanren would work, too)
Christopher Shank 2023-10-07 05:31:37

Thinking about syntax based on proximity and containment rather than adjacency.

Could you expand on the difference and/or tension here?

Alex McLean 2023-10-07 10:26:55

Christopher Shank Yes definitely! Conventional text-based programming languages are generally built on adjacency. This is a visual property in a way but I suppose a discrete one -- two things (characters, or words) are either adjacent or not -- so we don't tend to think of it as visuospatial.

Box-and-wire dataflow languages like max/pure data are built on connectedness, another discrete property. We tend to think of these as more visuospatial because you can arrange things how you like in a 2d arrangement, but this is all secondary notation rather than syntactical or semantic.

Proximity is where the arrangement enters the core language - things connect if they are proximal. This means you can assign additional meaning from how proximal things are. A really nice, successful example of this is the reactable. I'm not exploring this in the particular demo at the top of the thread, but think there are a lot of possibilities here. I explored this sort of thing some years ago with a haskell-based FRP front-end.

Containment I guess is actually a separate issue. In this demo I'm exploring just drawing around groups of glyphs. In text-based systems we use parenthesis for this.

One nice thing about exploring proximity is the possibilities for collaboration. The reactable is again a really nice case study for this - they made it circular so there's no way 'up' and people can collaborate by standing around it. Way ahead of its time really.

Pawel Ceranka 2023-10-06 15:33:49

We’re trying to put together a bunch of research tools in one place —web search, images search, all sorts of AI thingies.

It’s now possible to try some of this straight from the homepage —no login, no nothing.

Please check it out if it sounds interesting at all and as always comments appreciated 🙏

Here’s a little video for your viewing pleasure 📽 — have a great Friday and wonderful weekend!

Alex McLean 2023-10-06 17:42:41

Hmm, first try and it attributes my work to someone else as a 'key fact' 🤔

Alex McLean 2023-10-06 17:49:07

and the second try. my fault for searching up my own projects I guess..

Pawel Ceranka 2023-10-06 20:02:04

Yeah, it’s LLM based — it is hinted at during generation, but I guess it should be much clearer that’s the case and that it might be prone to some hallucination — sorry about that 🙏

Thanks for trying it out though!

Alex McLean 2023-10-06 20:12:27

I struggled to make it give a list of key facts that didn't contain lies, and of course they're all unsourced. Do you think LLMs have a place in research?

Alex McLean 2023-10-06 20:13:36

As it is, it seems dangerous to me.

Pawel Ceranka 2023-10-06 20:28:56

With this experiment we’re trying to learn if this kind of usage pattern i.e. generating interactive initial content for a spatial workspace would be an interesting thing to explore further and what are the trade-offs.

The quality of generated results could be improved even now, as e.g. the models we are using at the moment are not the best possible.

Before we determine, if it’s worth doubling down on this effort we’re trading of that quality for price and speed.

This generative aspect is mostly for getting you started on exploring some topic, there are other little tools and features in the app that should take you bit further.

There might even an argument made that an inclusion of some more sophisticated verification tool for the generated content would be beneficial—that is the kind of thing I’m interested to learn, so thank you!

I think the LLMs most likely are going to be widely used for research whether we like it or not. Given that I think it’s worth exploring the tools and interactions around them to find what’s useful, humane and fun 🙂

Alex McLean 2023-10-07 10:12:27

Yes sure LLMs could be useful for researchers, but I think labeling statistical jumbles as key facts isn't humane, and in the world of research will result in embarrassing retractions and probably legal issues. Sorry to be so down on this, I'm sure there are nice aspects to this work too, but I don't find it a nice experience to be presented with falsehoods like this.