Pragmatics first: does your blog have an RSS feed? I can't find one.

Your blog post reminds me of a podcast interview with Joscha Bach where he described a future in which humans will happily live together under the guidance of an AI superpower. He seemed to consider this a desirable future.

I’m using Jekyll, which I think can generate one. Thanks for the reminder. I love RSS.

Thanks for the reference. I’ll take a look at it. Always good to consider an alternative perspective.

Sounds like you enjoyed the piece. That’s great.

So many cool ideas in here, I like the tab/panes system too

I would love to know more about why you decided to build this!

Several reasons that added. I love observable and used it for several years. I think the mixed source with reactive runtime is genuinely an advancement. However, I keep hitting limitations with the iframe sandboxing and application users want something more recognizable or feel insecure piping data to a 3rd party domain. So I want a self hosted version that can be 3rd party self hosted. When the app is cloned you get a perfect copy that will always be like that, so it avoids things breaking over time problem. And it's a single file so it's easy to run, everyone has a browser. The goal is a substrate for forever software by avoiding tying it to servers or domains or native tooling.

Lovely! Hoping this becomes part of the platform someday 🤞

Can’t pretend to have read all of this of course. :) But I have been skimming through and think I understand the perspective you’re coming from. (I agree with many things that I read).

But I’m really skeptical about the 10x claims and I’m guessing many will be too. I don’t think the efficiency of building these kinds of systems is ever bottlenecked on these kinds of concerns. (I’ve worked on many different kinds of systems that fit you dis definition, all with their own different bottlenecks). I think also practically, many people will stop reading at that point. So if you want more readers, I’d recommend not putting that.

On a more content disagreement. I think you’ve got some work to do on the consistency issues.

To summarize our point so far: consistency issues arise when we attempt to respond to multiple calls in parallel, that is, instead of responding to one call and then to another call, and there is some shared data concerning both calls that is modified by one of the calls.

You can have inconsistency with complete serial calls if you just have a lossy communications medium. This is the most common problem I see with inconsistency in practice. X sends Y a message and unknown to X, it failed on Ys side. There are lots of results about consistency (CAP, CALM etc ) and practical things like Jepsen.

Writing-wise, I think your "Central thesis" needs to be right at the top, frankly in the first paragraph like they told us in school. I started skimming after about 5 seconds of abstract stage-setting, and in a less charitable mood I would have just stopped.

As for the thesis itself: I already agree with it, at least as the most basic level, but I'm not sure how actionable it is. I regret to say I don't have time to read the entire thing looking for new, actionable ideas. 🙃 To commit to reading something of that size, I'd need to see some evidence, right up front, that I'm going to see some interesting applications of the "focus on the data" principle.

Going on a bit of a tangent: sometimes I think these big manifesto thingies need two versions, one that focuses on convincing people that the problem exists in a particular way, and one (targeted at people who already agree about the problem) that focuses on a proposed solution. Doing both in one document is tricky at best.

Skimmed and scanned it like the others here, and have fitted it into my Cognitive Model of Prejudice already! Thing I personally disagree with is the imperative style (RPC, really) versus a declarative one. But, well, yes, a monumental work. Are there precursors to this that you iterated over, or did this all appear fully-formed just now on the internet here?

Also, what would it look like if you got everything you want and this were fully implemented? How much would stay the same and to what extent would everyone do things differently? What implementation code would be running and where? (personally I'm OK with the 10x claim btw)

I think your data-first perspective strongly leads to a declarative approach overall, not the call-response RPC style. You may need to go and climb a mountain and meditate on this fundamental dichotomy! Just use spreadsheets as a data point. I strongly believe and commend this to you as aligned to your broader perspective.

Thank you so much to the three of you for taking a look! Your feedback is super useful. I just made these changes:

• Add a link to the central point at the top, so you can skip the intro and jump there.

• Add a link at the top to a Programmers' Preface, for those that want a sort of "diff" between what they already know and what they might get from the treatise.

• Add links to each of the pillars just after the main thesis, so you can jump to them without scrolling.

I'll reply to your specific points in three different messages, to keep things a bit more organized.

Jimmy Miller

• Excellent observation about the 10x claim. I put it there to set the bar really high in terms of how much simplification I demand myself from this conceptual framework. My intent is to get people raising an eyebrow, rather than getting them drooling thinking of a payoff. But I acknowledge this is risky. Behind this (and I just put this in the Programmers' Preface) is that if many of us feel in our bones that easily 90% of software is pure accidental complexity (in the Brooks sense), then a 10x leap can come through by finding a way to drop this baggage with better tools (both conceptual and embodied).

• Concerning your point about bottlenecks, I am referring to the conceptual burden of designing and implementing a system, where human brains (not computer performance) seem to be the limiting factor of the speed of development. My personal experience with several of these processes tells me that the use of the wrong abstractions, and a certain unfounded fear of concrete data, is what kept many smart people going in circles around something that ended up being quite trivial once understood. I'd really like your take on what are the human bottlenecks you've experienced, because it's very likely I'm oversimplifying things here.

• Concerning consistency (and I'm grateful you made it all the way to the bottom!), I studied CAP and PACELC in detail as I wrote this section. Concerning your specific point, if X sends a message to Y and Y fails to process it without X knowing, this failure of consistency can be understood as an absence of stopping (or lack of transactionality). This is akin to the second non-parallel source of inconsistency I mention: Not checking for errors in multi-step sequences. ; the lack of acknowledgment from Y to X is to me exactly equivalent to a sequence happening in a single node that doesn't stop when an error happens in the middle. But I'm open to more challenge in that example, and if you can find other counterexamples to my point I'd really appreciate them. The whole section was an exercise in trying to think the problem of parallelism and consistency from first principles; in every instance of the problem I found, I could always narrow it down to calls being expanded simultaneously (even concurrently by a single processor) that touch related data. What was liberating to me of understanding like this is that the concept doesn't require partitioning, parallelism or concurrency, but it describes the problem at the core of them all.

Andrew F

• I just put a link at the top, and even mention the core thesis in the link itself. Thank you for skimming past that long slog of an intro.

• As for the applied cases, they are within each pillar. The pillars are alas quite lengthy, and I don't know how concrete they are. The most meaty one, in terms of solving concrete problems, is pillar 4 (logic), but probably 1 should come first.

• I've been thinking about your point of the two flavors of manifesto (one to convince that there's a problem, another to tell the convinced how can we solve it). I realize that I'm basically writing this for those who already feel there's a problem. Guess I'm in the right community, after all!

Duncan Cragg

• I wrote this over the last three months; previous technical writing I did was much more applied and definitely not an attempt at a complete conceptual framework. I got the idea for the main point and the five pillars in one go; the first draft, dated a couple of days after the original idea had different pillars but the embryonic core was there. While it may feel it's coming a bit out of nowhere, all of this was brewing in my head for a long time, and only made possible by myriad influences from others (for example, a lot of Rich Hickey and Joe Armstrong in the last few months).

• As for how it'd look if these principles were made into a tool, you put me in the spot! What I'll work on next is cell, which is a programming system. I plan on building it based on the principles of TODIS, and fully expecting that I'll have to perform surgery of uncertain extent in TODIS based on my experiences inside the trenches of cell. I'll be very happy to share some progress in cell when I have something, but for now it's absolute vaporware. Your questions about how things would look like if these ideas were implemented, how would people be working differently with software, I'll have foremost in my mind when I design cell. I really appreciate you stating them.

• As for calls being imperative, I'd love to analyze this further with you. In the framework, there's nothing inherently declarative or imperative about a call. If anything, a call, in the sense of communication with intent, it's really a "what". The "how" lies in the implementation of the call (the logic, pillar 4); whereas the outside part of the call is an "interface", pillar 5. At the very end of pillar 4 I spout:

Microcode could be seen as a declarative interface to a CPU ("I tell the CPU what operation I want and let the CPU actually manage its own logic gates"). Similarly, a line of a very high level program might be seen as imperative ("give me all the rows from this database where the creation time is less than three seconds ago"). I propose that we eliminate this distinction

• In other words, at every level the outside of the call is declarative and its logic is imperative. There's no hard absolute line that says "above this, it's all declarative"; our understanding of it is self-similar, fractal, working at every scale in the same way.

• Rather than try to convince you, I'd really like to understand why/how you see that a model based on calls is inherently imperative; and perhaps more importantly, whether declarative operations (when they are requested, not when they are implemented) cannot be modelled themselves as calls. I promise to keep an open mind!

• I just discovered your work on Onex. It seems we're both in love with the spreadsheet! Cell too is a programming model based on the spreadsheet. Really cool that you're working on that direction!

Everyone who joined in this thread, thank you again for your brain cycles and kind words. They mean a lot.

Concerning your point about bottlenecks, I am referring to the conceptual burden of designing and implementing a system, where human brains (not computer performance) seem to be the limiting factor of the speed of development.

All the systems I've worked on where human brains were the bottleneck, I don't think would get 10x faster with this (as far as I understand it process). But all the system I can think of where human brains are the bottleneck might not meet your definition of DIS (compilers for example).

Behind this (and I just put this in the Programmers' Preface) is that if many of us feel in our bones that easily 90% of software is pure accidental complexity (in the Brooks sense), then a 10x leap can come through by finding a way to drop this baggage with better tools (both conceptual and embodied).

My reading of Brooks (see the episode for more) is that he was concerned with industrial scale 10x improvement. That companies can actually produce software 10x faster. I guess that's what I had in mind.

This is akin to the second non-parallel source of inconsistency I mention: Not checking for errors in multi-step sequences.; the lack of acknowledgment from Y to X is to me exactly equivalent to a sequence happening in a single node that doesn't stop when an error happens in the middle.

The problem is, in a lossy communications channel, it is impossible to check for errors. Acknowledgements don't solve the problem. Because how do I know the person who sent me the message got my acknowledgement? The two generals problem illustrates this well.

But it goes deeper than that two. Being distributed makes a huge difference in the kinds of failures you can have. For example, it has been proven that it is impossible to tell the difference between a node taking too long and a node that has crashed if you have even one faulty node

I don't want this all to come across as over negative. I love people creating these high minded general approaches. Even if I don't agree with any of them in particular, I think it is good to think through and have this kind of thinking for people to contrast with. If you haven't seen his work, I'd definitely recommend some talks from Peter Alvaro. He's got one on writing software with an order of magnitude less code using a datalog derivative. Shares some of the squashing of call vs remote that you are interesting in. (But with explicit time.) youtube.com/watch?v=R2Aa4PivG0g and he's got another on persistent memory and a globally singular dataspace (The whole world has one dataspace) youtube.com/watch?v=aIqamaY5pzM

Having the links definitely makes it little more manageable to find what I want to know. The "Pillars" are definitely the interesting part that makes the rest into something other than rehashing the same abstract problems we're all thinking about. (And they're pretty similar to a lot of my thoughts, so that's neat.) I would suggest putting a preview of what they are somewhere more prominent in the document, up with the thesis statement (which TBH I still think needs to be actually present above the fold, not just linked) so it's clear that there is some real technical meat there.

FWIW my ideas for the common data format start the other way around, as a crystallization of what you might label call and response, more abstractly crystallization of control flow (something something algebraic data types and algebraic effects). But it's not even as concrete yet as your idea here, so...

I think it may take a while to digest all the cell stuff and the other bits. (Also, I haven't actually earned an acknowledgement yet, but you do inspire me to contribute more!) 🤗

Initial thought: cell's "quadrivium: language, database, service and interface" map onto my own ONR, ONN, ONP, ONT: language, core database, p2p networking, i/o. Are you stealing my ideas??? 🤣

Onex vs Cell: I don't have number as a separate type, just string or symbol representations of number. This is biting me a bit in my string interning algo, as each tiny change in a number creates a new string in the pool. I sketched out an alternative for numbers but binned it. Even so, this is all about non-functional requirements - efficiency - not about the higher level model.

still in the weeds, and feeling like I'm hijacking this thread so this will be my last comment for now: "As for null , it can be represented by the absence of a certain key in a hash; but inside a list, it has to be represented with something else, such as the text null or an empty string." - I have two special concepts for unspecified values: unknown (could be anything but isn't set yet) and nothing (known-to-be-absent or is definitely not going to be set). Or not available vs not applicable. Setting a hash or map property to nothing deletes that property. setting a list value to nothing deletes that entry in the list.

Hi! Back again. On the "calls being Declarative, implementation being Imperative" topic: are you saying all APIs wrapping implementations are Declarative? That's not what most mean by Declarative! This is a good article to read in this area: stackoverflow.com/questions/602444/functional-declarative-and-imperative-programming/8357604

The common (non-academic) understanding of Declarative is that you declare a bunch of relationships in any order and then apply those to real data to get more real data. Every time you ask the question, and regardless of the order that your declarations are applied, the answer is the same. This so far means Declarative isn't Turing Complete, but to achieve that you simply circulate around your answers or outputs back into your questions or inputs.

It's a little more declarative in detail, where a "call" seems to enforce a relationship between parts of the "dataspace". That said it looks like a pretty thin wrapper over just calling a procedure again with the new arguments. Unless I missed something big.

Jimmy Miller

First of all, you're not coming off as negative at all. Your arguments are thoughtful and got me thinking. The fact that you see things quite differently makes your perspective all the more enriching to me. So, truly, thank you for that!

Just a few points replying to yours, and honestly no need to re-reply unless you really want to!

• I do consider compilers to be a DIS. They don't store data, but they surely communicate it and transform it. I'd actually love to apply this approach to a compiler, and will soon have to do this in one way or other when I implement cell.

• I indeed recently found out about unavailability being indistinguishable (or rather, at the end of the continuum) of latency. And it's quite elegant to see it like that! And indeed, with distributed systems you have an eternal recurse: how do you know if the acknowledgment of the acknowledgment of the acknowledgment reaches its destination? I think this can be thought of as a probability: each call (not even each side) has an error threshold, and above it it considers the operation to be successful (and failed otherwise). I think that distributed systems only makes the probability of error higher (and the probability of not finding about it also higher), but that could still happen in a single system (solar flares?). Your point has uncovered this angle for me: the assumption of dealing with certainties (rather than probabilities) of knowing what's happening.

• I'll check the podcast episode and also the work of Peter Alvaro, thank you for the recommendation!

Andrew F

• My fear about putting the pillars "up there" is that the document will be too geared towards programmers; I still have the hope that non-programmers that are interested enough in information systems might get some value from that intro that, from a programmer's perspective, seems to be a "fish describing the water". I could be utterly wrong, though.

• I'm interested in your take that calls come first, data comes afterwards. If you have something in writing about it (or want to post about it here), I'd gladly take a look. To me, I feel that you first have to have the chemical elements in order to later get the unicellular organisms. If data are the elements, the coming to life comes afterrwards. Then, you might reach a situation where most of the matter in an area is actually within live organisms, and it flows according to their own logic.

Duncan Cragg

• You've squarely earned an acknowledgment, in my eyes. You're the fourth human to give me feedback about TODIS!

• I'm not stealing your work in Onex, yet. Now that you linked it and I know about it, I might 😄. But always giving due credit. And cell will be fully open source and public domain.

• Concerning the fourfold mapping, amazing! It means we're seeing convergence; we were working separately on similar problems and are finding that we need the same things. To me, that's very healthy; it indicates the problem domain is not a solipsistic figment of individual imaginations, but something rather more objective. As for interface, I see it as an "interface maker", rather than i/o. But if you were referring to interfaces by i/o, then the mapping is 100% the same.

• To me, number needs to be its own type, conceptually. The operations on them are very different from the ones we do on text. And I'm in love with the notion that from the two single data types (number and text) you can extend to the two multiple data types (list and hash), where numbers are the keys of list and texts are the keys of hash. It's just too elegant to pass up. Now, ask me in some time when I'm in the cauldron of implementation. But I'm a sucker for conceptual integrity.

• Concerning null or undefined, I really don't want to add any values that are not part of the four base types. That's why I narrowed down on the empty text as my only null-like value. It's still a convention, which I like, because it gives others the possibility of picking something else, as long as they make it a value. The other thing I want to do concerning this (warning: vaporware) is to make references to nested objects that don't resolve to anything respond with an empty text, rather than throwing an error. Redis has this attitude towards structures that don't exist and I think it's the mother lode.

• On declarative vs imperative, my contention is that you can put any operation in terms of a call, and that you can consider really anything as declarative or imperative (I know, bring the kindling and the stake, I'll even help you light it up). This is not a troll, though, I do mean it. Let's take a declarative call, like a SQL statement (is that a good example? If you have another one, please do send it to see if I can still make a point about it). Let's say that the SQL statement has some order required to it, but it's safe to say that you're not sending a search algorithm over the wire that the database will execute for you. Still, you're sending a message to a destination (that's what I consider a call, by the way). As a response, you obtain the result to your query. Is that declarative? It can be. But consider now me sending an opcode. I'm also sending a message (this time, to the CPU), saying that it should perform an operation. Now, am I telling the CPU how to organize its logic gates to achieve that result? Hardly. Whether the CPU uses complex logic, or microcode, I haven't specified. All I care about is that I get my response. The lack of ordering in what is normally considered a declarative call is simply a convention, in the same way that the order of arguments you send to an opcode is also a convention. It is a way to express what you want; the destination, which processes (expands) your call into a response, is in charge of that how. And even very declarative layers, like DSLs (including SQL) are actually quite structured in how they take their inputs. Even more unstructured/higher level paradigms (like LLMs) an still be seen in the call/response model. Does this make any sense?

• Clearly there's a difference between a SQL query and an opcode. The former is built with the latter. SQL is more high level than opcodes, but that doesn't preclude us from using the same model to describe each of these operations as calls, and to see all the commonalities between them (destination, message, response and expansion).

It's really hard for a document to serve two such different audiences, and frankly I don't think non-programmers are going to be reading this any time soon.

I don't have anything written about my ideas yet. I can say it's inspired in part by tagless-final encodings of data, where the data is basically represented by a function that traverses/outputs the data into an interpreter. I can't find a good concise intro about that either at the moment, but if you want to dive in the deep end: okmij.org/ftp/tagless-final (very heavy on the DSL interpretation rather than as data, but hopefully you can see how data could be a specialized DSL if you squint).

On "stealing" - I did a big fat emoji to indicate I was joking! I actually agree with everyone taking everyone else's IP (apart from trade marks) - that's how humankind made progress throughout our evolution and civilisation, until recently when it was all stolen, from the common good, by mega global corporations.

I think we should move on from the Declarative rabbit hole, then, as you basically repeated what you said before so we're not getting further on! But! There is hope for a productive follow-on, as you talk in terms of reactivity and repeating calls when things change, like in a spreadsheet. Spreadsheets are Declarative, so that makes me happy. Also, you mention HTTP calls, and when idempotent, they have a Declarative vibe (I see REST as a Declarative architectural style).

This work all has similiarities to Substrates, don't know if you were aware of that? ex-situ.lri.fr/content/8-teaching/4-fundamentals-of-situated-interaction/2-handouts/substrates-tr2017.pdf Also there's some commonality to TBL's SOLID.

Where do "applications" fit in this? My preferred answer would be "nowhere, there are none!"

cc @Elliot @chee @Chet Corcos Mariano Guerra @William Martin @Daniel Sosebee

nice! are you presenting it in the next FoC virtual meetup? 🙂

Hmm last time I had shown the prompt editor, I had some really great feedback that I should somehow demonstrate the development cycle using the dev tool so I kinda wanna think about how I can demonstrate that with this