Charts and diagrams use visual structure to convey meaning. For example, spacing and alignment groups data, spatial proximity links labels to elements. But this visual structure is typically invisible to screen readers.
thread: bsky.app/profile/jonathanzong.com/post/3m3nwszhtrk2k
Graphical representations—such as charts and diagrams—have a visual structure that communicates the relationship between visual elements. For instance, we might consider two elements to be connected when there is a line or arrow between them, or for there to be a part-to-whole relationship when one element is contained within the other. Yet, existing screen reader solutions rarely surface this structure for blind and low-vision readers. Recent approaches explore hierarchical trees or adjacency graphs, but these structures capture only parts of the visual structure—containment or direct connections, respectively. In response, we present Benthic, a system that supports perceptually congruent screen reader structures, which align screen reader navigation with a graphic’s visual structure. Benthic models graphical representations as hypergraphs: a relaxed tree structure that allows a single hyperedge to connect a parent to a set of children nodes. In doing so, Benthic is able to capture both hierarchical and adjacent visual relationships in a manner that is domain-agnostic and enables fluid (i.e., concise and reversible) traversal. To evaluate Benthic, we conducted a study with 15 blind participants who were asked to explore two kinds of graphical representations that have previously been studied with sighted readers. We find that Benthic’s perceptual congruence enabled flexible, goal-driven exploration and supported participants in building a clear understanding of each diagram’s structure.
Sounds very congruent with siddharthaprasad.com/unpublished/pgnk-lightweight-diagramming.pdf!
thanks for sharing! I'm reminded of David Harel's higraphs and the usefulness of describing topological relationships
Kartik Agaram Yes! I must sit down with it before I can language the connection. Glad to see Gestalt principles centered in both.
Christopher Shank These remind me of dataflow graphs, but really nicely drawn by hand, with a lot of principled decisions going into the encapsulation (e.g. figure 21). I feel like we're all still building the future OP saw 37 years ago:
We are entirely convinced the future is “visual.” We believe that in the next few years many more of our daily technical and scientific chores will be carried out visually, and graphical facilities will be far better and cheaper than today’s. The languages and approaches we shall be using in doing so will not be merely iconic in nature (e.g., using the picture of a trash can to denote garbage collection), but inherently diagrammatic in a conceptual way [...]
I looked up Harel's Wikipedia page, which had this history of UML added. I feel that Star's boundary objects (1989) are another key example of state charts, with their own parallel lineage.
commons.wikimedia.org/wiki/File:OO_Modeling_languages_history.svg
@Jasmine Otto Interesting quote! So why didn't this happen? My hypothesis (untested) is that our computational models are semantically too complex for being encoded visually. Visualizations as partial views on something's semantics have worked out very well, but visuals as the only (and thus complete and precise) representation has not, except for semantically simple contexts such as the popular node-and-wire representations for dataflow.
Konrad Hinsen Absolutely - I don't think that's a no-go, however; 'visualization as partial views' are super effective for communication and teaching within rich domains, which have their own jargon ('compression') for high semantic complexity. But the jargon makes it hard to share models with each other; within-STEM is bad enough, and that's after a 4-year onramp in undergrad. (E.g. I think of compilers / operating systems as the keystone / shibboleth for CS undergraduate, building upon data structures and algorithms.)
But it's not the fault of non-technical stakeholders that they don't share this language. Moreover, a well-designed visual encoding of the computational model is ideal for exfiltrating it from domain jargon. Boundary objects aren't "complete and precise" representations of the system by design, they're just good enough that coordination is possible in these high technical-debt situations where ordinary conversation likes to spin out.
I think the 'partial view' formulation is very apt. It reminds me of textbooks that have a flowchart showing dependencies between chapters, that are meant to be read by random-access as needed.
Visualizations and visual GUI controls as boundary objects looks like a good perspective to take!
This is probably off-topic, but this discussion is tickling my neurons along a different dimension: to me issues of VPLs, DPLs and TPLs avoid the elephant in the room - how to express complicated solutions in a way that is convenient for humans to grok. What we’re currently doing - imports, over-use of the synchronous sequential design method, over-use of static type checking, over-use of the functional paradigm, bloat, forced insertion of concepts that are clearly out of the sweet spots of host paradigms, etc. - are red flags
