One more spreadsheet experiment: what if it wasn't just the cells themselves that could have formulas but things like cell styling too?

ezhik.jp/spreadsheet-2

I assume you consider "layering" to be a bad thing. Totally valid criticism. But there are drawbacks in the other direction as well:

• If you generate code from diagrams your diagrams now have to do double duty. They have to be a source of truth and also hide detail.
• You tend to be stuck with one totalizing way of viewing a large program. There are no arrows in the second picture because it's a different kind of picture . Glossary rather than blueprint. To support multiple viewpoints in your approach requires checking them for inconsistencies, and showing inconsistencies in a nice way to the programmer so the programmer can understand what the compiler has found.

These are good directions for research. But my doctrine is that I'm trying to do as little research as possible. My focus isn't to make everything perfectly clean and pleasant to use. My focus is to help people take control of their computers with the mature technology we have today.

This is why I'm adamantly not creating a new programming language.

To elaborate on "glossary rather than blueprint," we often hit points in programs where you can't stage learning. You can't learn A without knowing of the existence of B, and vice versa. In these situations it helps to have just a todo list. You need to learn these things. It won't make sense at the start, but here's a bound on what you don't know yet.

I hope this conveys that my primary audience isn't the author of a program. It's the reader. If the author has to do some extra work to engender trust, I'm ok with that. A program will always consist of more than code. Keeping the comments up to date is part of fostering the reader's trust. You can't blanket distrust all comments, I think you miss out that way. Different people put in different levels of attention to them. I'm sure my projects are hard for others to understand, but they seldom contain outright falsehoods in documentation.

I am pretty much pursuing the same goals as Kartik Agaram, just in a different setting. In particular, I use plain old code plus different views to make the system more understandable. But I am also interested in using diagrams as the ground truth, replacing code, for parts of systems.

The big difference it makes is about changing things. If there is an author who makes an effort to facilitate the work of the reader, that also means that the reader has less of a chance to become an equal to the author. And that means lower conviviality.

If your ground truth is a diagram, then you know that there is nothing else but the diagram and its semantics that you need to understand. No view can make such a promise, it's always a projection. Of course, that's also what makes this idea hard to realize. I am not aware of much work on formal semantics of diagrams, nor of their understandability.

I assume you consider "layering" to be a bad thing. ...

Actually, it's exactly the opposite. I think that layering is vital . Our current PLs are just forms of fancified assembler. This is a metaphor that is at least as tired as that of desktops. The tired metaphor of assembler leads to overly-rigid beliefs about programming.

Ah, thanks for that clarification. Ok, now I fit it right with your past writings. So you think it's a step in your direction, but it doesn't go far enough. Totally agreed.

Konrad Hinsen I'd suggest that overly formal semantics can also drive people away. Which is not to say my current level of formality is precisely the sweet spot 😆 I do design my data structures as tables containing tables containing strings and numbers, etc. That likely drives some people away.

But one major alternative to formal semantic guarantees is to keep things simple enough that the universe of possible things to understand is not very large.

Paul Tarvydas I was thinking about your layering of "computational circuits" (which is how I tend to think of your approach) yesterday when I was looking at my own notes on code and data hierarchies, and I was wondering how data fits into your scheme.

As long as you only have a fixed set of atomic types (numbers, strings, ...) in your queues, everything is fine. But how about structured data? Every component must then include a definition of the data it expects as input or provides as output. It must also pass this data on to its subcomponents, but it may not expose their data type definitions at its own interface. This doesn't sound impossible, but I am not aware of any of today's programming language that would support such a scheme without laborious and possibly inefficient data conversions.

Konrad Hinsen This is probably a dissatisfying answer... How do you do this kind of thing across vast distances? Say, a machine in Toronto, Canada speaking to a machine in Paris, France? I'm of the opinion that you don't do this. Uncomfortably, the answer is something like JSON. The concepts of Programming Languages and Data Structures were invented to appease the biases of the 1950s, wherein "efficiency" was vital and time-sharing was vital and memory-sharing was vital and memory-structuring was vital. Today's hardware is entirely different from that of the 1950s. Today's problem is how to program DPUs (Distributed Programming Units) instead of CPUs. Thinking along those lines leads to completely different answers than what we're used to (in fact, I currently think that we need to deprecate O/Ss, PLs, Programming Using the SICP Method (PUSM), data structures, etc. [Aside: note that the current trend is to offload real work from CPUs onto GPUs, anyway]). Of course, we can't just erase everything we've got and start again. The giants of the 1950s/60s/70s/80s built on what they had (vacuum tubes, transistors) and came up with something new (they didn't just glom stuff into what they had, they didn't just make better vacuum tubes by incrementally tweaking the concepts). I'm nipping away at reimagining our biases, for example, simply using boxes and arrows and HTML elements as syntactic elements for "programming languages for the 2000s" instead of just using 7-bit characters. I'm using what we've got and trying to think about it differently. We know a lot more about programming techniques than we did in 1950/60/70/80 - can we just rearrange what we've got and build something fresh? So, you're right, I don't know how to handle complex data structures, but, I don't care.

Konrad Hinsen Kartik Agaram Let’s see if I can say this in a way that seems more practical. The following is about as short as I can make it for now...Composing Multi-Node Systems programmingsimplicity.substack.com/p/composing-multi-node-systems?r=1egdky

I think we have wildly different assumptions of scale in mind. I don't care much about zillions of nodes, shared memory, cache coherence, distributing computation across a room let alone timezones. They're valid use cases, but designing for them feels like the tail wagging the dog, to my mind. A single computer today can do so much in a single thread. I'd always choose the programming model that provides user-defined data types over the one that provides nice concurrency properties. So your model has some nice-to-have properties, but the lack of user-defined types feels like a deal-breaker.

Thanks Paul Tarvydas for an exhaustive answer to an interesting question - but it was not the question I have. I am not so much concerned with data representation for storage and transfer, but with data models or schemas, and in particular with ensuring that there are no misinterpretations at the interface between two components.

In my corner of the computing universe, data is the primary concern of most people. For example, I work a lot with protein structures, which are structured collections of collections of atoms plus lots of annotations at all levels of this tree. What I care about is being able to feed such data structures into processing pipelines, storing those that come out of it, and being reasonably certain that all annotations have been interpreted correctly.

Konrad Hinsen Kartik Agaram I argue that this is the same problem, but looked at from a different end of the telescope. The tail-wagging-the-dog approach is to nip away at the problem incrementally. ... more to come ...

Yes, this seems worth digging into more. Is your "fewer functions" quest dual to "lots of concurrency," so that the proposal is you can never have compound data structures without the ability to serialize them and so on, and those considerations are first class to how they're designed. To me serialization seems a subordinate consideration to say Lua's syntax for numbers, strings and table literals. I don't need to think about serialization because anything can be converted to say JSON after the fact. It seems to be an independent question and "good design" is grabbing opportunities to decouple design considerations at every opportunity.

In my courses on computational science, I insist on the difference between "data model" and "data representation". JSON is a representation, and as Kartik Agaram has pointed out, any data can be serialized to JSON (or XML, or s-expressions, or ...). Choice of representation is a matter of convenience and efficiency. I am much more interested in the semantics part, the data model. I don't see (for now) how it could be part of the same problem.

I like your wish list! And your so-far-documented design decisions look reasonable. At least in theory, it takes an implementation to find out how they work in practice.

Wondering: have you compared your design in detail with existing systems? I think that would help in understanding what exactly you are missing, and to what degree such features can be added without creating problems elsewhere.

For example, at first sight your with list and design decisions has a huge overlap with Common Lisp. What I cannot judge easily is how difficult it would be to morph Common Lisp towards full alignment with your goals. Given how well understood Common Lisp is (both good and bad aspects), a detailed comparison should be helpful.

For for staged compilation and modules, Racket has probably the most sophisticated approach today, but it may well be more sophisticated that you would like for your system, given that Racket allows each module to be written in a different language.

Ah, I forgot to add the problems from scattered-thoughts.net/writing/there-are-no-strings-on-me to the 'legible' side of the wishlist. I'm explicitly trying to avoid the lisp/smalltalk model where loading code means executing a series of side-effects against a mutable environment.

In particular I don't want to allow things like:

if ((new Date()).getDay() == 1) { 

    Date.prototype.getDay = function() { return "funday" } 

}

I added:

• code-based rather than image-based - the code running now is the code that you loaded, nobody can sneak in and change it at runtime, you can understand the system by reading the code
• no mutable environments
• file/module/expr load order is not observable
• no side-effects at load-time, no life before main
• no non-determism at load-time - loading the same code always produces the same program
• cf there are no strings on me

Also under performance:

• reflection and meta-programming are predictably staged away

The closest existing languages are zig and julia, which both feature type-checking after specialization and staging via specialization (values lifted to types) rather than quoting.

The latter means you can understand the program via its dynamic semantics. Type-checking and staging only affect performance - they can't change the result of the program.

But neither of those languages have the restriction to value semantics, which complicates the module system.

docs.racket-lang.org/reference/eval-model.html#(part._separate-compilation) in particular is very unsatisfying. It means you can't give describe the meaning of a racket program without resorting to talking about a lot of compiler machinery.

I'm less familiar with common lisp, but searching for "common lisp load order" lead to reddit.com/r/Common_Lisp/comments/xt7jn4/how_to_build_a_proper_loading_order_from_asdf, which is about the experience I expected.

you wouldn't be able to reproduce that way; file compilation has different semantics than repl interaction (the latter happens strictly at "load" or "execute" time, while the former happens partially at "compile" time)

FWIW: ASDF is not part of CL. It is a program written in CL that defines a "declarative" DSL with lots of hoary edge cases for determining dependences and loading the minimum number of files. It was invented back when machines ran slowly enough that Makefile seemed like a good idea for minimizing CPU cycles. The basic load function defined by the CL language is (LOAD ...) which you can run from the REPL. One can forego the use of ASDF and simply write a .lisp file with a bunch of (LOAD ...)s in it. It is my understanding that modern CL REPLs actually compile every command, then run the compiled command. In essence, if you type (LOAD "filename") at the REPL, the REPL compiles the command into a small binary program, then runs it, actually running the LOAD command built into the Lisp language. ASDF just generates a bunch of (LOAD ...) commands. The definitive reference for CL is CLHS, but it reads like a ton of bricks (it helps to have a degree in Law). If I had to re-learn Lisp, I would probably start with Practical Common Lisp

So still the same model I was complaining about in scattered-thoughts.net/writing/there-are-no-strings-on-me.

Most live systems gain their interactivity from mutable environments and late binding. Code loading is an imperative process that can cause arbitrary side effects. The behaviour of the system can depend on what order code was loaded in, and when. There isn't even any guarantee that loading the same code in the same order will reproduce the same system.