Oh this is awesome! Common Lisp is definitely on my very short list of suitable platforms for Convivial/Freewheeling computation. So I'm very interested in contributing.

I'm still working my way through the demos. Some initial impressions:

• Are you really running Common Lisp within the browser?!
• Alt-click is kinda unreliable. I tried alt-clicking on the close/reload/maximize buttons on the top-right, and it was unclear what I was getting back in response. On one occasion I got a blank frame with just an outline and nothing else. So far I haven't gotten it to show me anything useful.
• In the Navigating a HyperDoc pane, in the Computed text section, it would be good to show an example of computed text with the dashed underlines. Maybe have 7*6 compute to 42.
• Meta lesson from the previous bullet: would it be possible to create publicly accessible hyperlinks to specific sections in a HyperDoc? 🙂

Continuing to read..

• The way the horizontal scrollbar highlights background when I hover on it, I worry that there are 2 nested scrollbars one on top of another. Maybe just stop highlighting the background on hover?
• In Influenza epidemics in France , could you link to a definition for relative incidence ? Should it go to en.wikipedia.org/wiki/Relative_risk?

• What would it take to enable the Playground?

• In the table ari-incidence-france , I wish each checkbox/line would have a consistent color.

Common Lisp (SBCL) is running on the server. The communication with the browser happens via a WebSocket, managed by CLOG. But Common Lisp in the browser is coming. Daniel Kochmański (current ECL maintainer) has a proof-of-concept of a compiler to WASM, and just announced that he received a grant to continue working on it.

Alt-click works on all items in the title bar of a pane, and on the tabs representing the views. It does weird things indeed on the close button - I hadn't imagined somewhat might try 😉

Computed text: good point. You can see one at hyperdoc.khinsen.net/377F4-influenza-epidemics-in-france, at the end of the first paragraph.

URLs for pages/sections: as my example above shows, there are persistent links to sections. In the "URL" view. I could do those for pages as well.

Ah cool. How can I find the URL for a section?

I don't see the scrollbar issue (Firefox). And I fear that my CSS competence is insufficient to deal with such issues!

Relative incidence is just cases divided by population. Not sure there is something obvious to link to. I should ideally just compute it, the code would then be the unambiguous definition.

Enabling or disabling the playground is done via a flag when starting the server (development=nil).

Colors: good point. codeberg.org/khinsen/ari-incidence-france/issues/1

Ah cool. How can I find the URL for a section?

In the URL tab on class hyperdoc. Assuming that's what you mean by section.

Enabling or disabling the playground is done via a flag when starting the server (development=nil).

I imagine there are security issues involved? Maybe just tweak the message to say it will be available when running locally, and point readers at the repo.

Yes, it's disabled because otherwise people could run arbitrary Lisp code on the server. The message should indeed be more explicit!

codeberg.org/khinsen/hyperdoc/issues/1

BTW, thanks for all the great feedback!

I'm definitely not seeing any change when I alt-click on tabs representing views.. oh, I just reloaded the tab and now I see it working. So there's some flakiness somewhere. At some point in a session it very commonly stops working for me, to the extent that it seemed totally broken across multiple sessions until now. I'm using Waterfox, and will also test on Firefox in a bit.

How can I install HyperDoc's dependencies locally? I don't really understand asdf or QuickLisp yet, so the command in the Readme doesn't work for me.

Common Lisp software installation comes in three layers:

• The language itself has (load ...) for reading in a file. And nothing else (well, there's defsystem, but it's basically useless today).
• ASDF is the de-facto code management layer that nowadays is shipped with all CL implementations. Its unit of software is the 'system', defined in terms of code files and dependencies in a system definition file (extension .asd). That's how all Lisp code is packaged nowadays. If you manage to track down the repositories of all dependencies, recursively, you can clone them into $HOME/common-lisp and you are ready to go: (require 'asdf) and then (asdf:load-system "hyperdoc-demo") .
• Software catalogs and package managers take care of downloading dependencies from the right places. There are the system-wide ones (Debian, Guix, ...), but also a few Lisp-specific ones, out of which Quicklisp is by far the most popular. It's very easy to install, and mostly just works. But its default software catalog is updated only about twice per year. If you want bleeding-edge code (such as mine), you have to add a second catalog, Ultralisp, which is easy to add as well. If you have those two, and followed their advice of activating Quicklisp on startup (in $HOME/.sbclrc for SBCL), then (ql:quickload "hyperdoc-demo") will download and load everything you need.

I provide instructions for Quicklisp (which I use on my old Mac) and for Guix (which I used under Linux). I have tested my system with SBCL and ECL, the two Lisp implementations well supported by Guix.

Philosophical note: Common Lisp is fortunate to have ASDF separate from Quicklisp and other package managers. The fusion of their Python or JS equivalents makes it a pain to assemble polyglot software. With Lisp, any package manager can delegate the Lisp-specific stuff to ASDF. And Lisp developers don't have to know about anything else than ASDF.

I provide instructions for Quicklisp

Oh did I miss this? I don't see where you do.

(I do have QuickLisp setup. I can be dangerous with it even though I have no understanding of it.)

codeberg.org/khinsen/hyperdoc-demo

The HyperDoc demo and all its dependencies are on Ultralisp. If you use Quicklisp and Ultralisp, you can run the Web server using:

sbcl --disable-debugger \

     --eval '(require :asdf)' \

     --eval '(asdf:load-system "hyperdoc-demo")' \

     --eval '(hyperdoc-demo:serve :port 9000)' \

     --eval '(sleep most-positive-fixnum)'

Ah thanks. I was looking at codeberg.org/khinsen/hyperdoc

I just updated the server with various improvements as discussed above!

Hoo-boy, I just noticed installing Ultralisp requires glorified curl |sh on a http rather than https URL.. I might wait a bit to try running this.. definitely on my radar, though.

It's not quite curl | sh . The closest moral equivalent I see is using npm . Installing Ultralisp is low-risk, but there's a subsequent risk in installing packages from Ultralisp, which is a mostly uncurated software catalog. http then seems like a minor additional risk. My understanding is that it's due to Quicklisp not handling encryption for some reason. So if it's http you are worrying about, you shouldn't be using Quicklisp at all. Which is one reason why my main Lisp development environment is managed by Guix.

Ugh, you're right. I see now that the transcripts on the Quicklisp frontpage involve getting from http URLs. This is going to require rethinking my Quicklisp setup..

(I personally don't see much difference between curl |sh and the npm eco-system. But they do seem to benefit some from at least pervasively using https. I like the idea of supporting http and tend to be on balance slightly against https. But there have been too many cases of intermediaries inserting data into payloads to justify downloading code over http , IMO.

Minor story: back when I launched lines.love in Mar 2022, akkartik.name used to be served over just http . So I hosted my download link at tilde.club. When I realized certbot was now push-button easy, I added https as well, but made it a point to keep http working. But now I notice akkartik.name redirects to akkartik.name 😬 Gotta debug that some time..)

Zooming out on my security stance:

• I care very much about minimizing dependencies and knowing where I get them from. I mistrust overly large and popular vendors, as well as tiny and anonymous vendors. I want to know more about who the people are behind a program or library.
• Given that concern about people, I also care about making sure random large/popular or tiny/anonymous folks can't get between me and the people I choose to enter into relationships with.

I am ambiguous about http vs. https as well. Bot-in-the-middle attacks are real enough to worry about downloading code via http . But https implies serious dependencies on Big Tech. Let's Encrypt is what makes https accessible to most of us small players. But it's backed by the big ones, and they can pull out any time once http support in browsers and other tools starts to disappear or become unusable.

Guix (and others) provide a better solution to code trust: it's not based on the source, but on cryptographic verification of the code itself. Nothing fancier than checksums, but applied systematically across the whole dependency chain, including build scripts.

Of course, no technology is more secure than the people who run it. It is conceivable that someone puts malware into popular code (the xz story), and that can happen at all levels, including Guix' build scripts. Perhaps the best protection of the Common Lisp ecosystem is its small size: it's simply not attractive as a medium to spread malware.

This pitch for Frest resonates with me!

Lots more to say. As an abstraction for developers, for a start.

I've thought about this a lot without progress. It's still not clear to me it is solvable.

Sure, it’s solvable. You use the power of the relational database.

You tag every row in every table with:

• a “at” timestamp; and
• a model id

Separately, using other tables, you track how models relate to each other. This one was branched in this way from this one.

You’ll note that you can imagine several different ways for model branches to work. Assume model B is branched from model A. What several options wrt how changes in A are seen in B:

• B’s view of A is frozen at the time it was branched, and no other changes in A are visible in B;
• B sees updates to A, but its own values overrides the values in A; or
• B sees updates to A, and values in A override values in B

If you reflect on that for a bit, you realise sometimes you need one, sometimes another. In my blog, I discuss what that looks like.

Note that whatever you have, it can all be resolved with relational queries. Indeed, you can have something akin to Postgres’s Row Level Security, which is an automatic addition to any query. It would not actually be difficult to instrument a relational database so that in order to query it, you have to supply a model value, and the tables automatically acquire the extra criteria they need to only show the values from that model and its antecedents.

Are models the same as relational db tables?

Oh I see, each row has a model id.

I don't see any reason to say "sure" about this 😆 You're recommending saving a multiple of data for multiple forks of models. That seems like the big source of risk here. I'm glad you're working to de-risk it.

The model ids pick out entirely separate versions of the same database, all with the same schema. Each one is a model.

Then, you have merge operations to support branch-merge semantics.

You mention risk. You can handle authorisation with the same Row Level Security semantics. This means you express relationally which roles have which permissions with which models, and enforce that using RLS.

You would also provide functions on models. So models can be “like this one, except I get to insert code before/after the values are found to change those values in whatever way I see fit”. IOW, you can have virtual models. A virtual model would be how you’d handle things like merging multiple models in a lazy manner.

When I said "risk", I was thinking only about performance issues. How large the database grows, how long it takes to clone one model from another, or how long it takes to query a specific model alongside its antecedent models (which is how I interpret "virtual models"). No matter how fast these operations are, they scale in proportion to the number of models one has, a number you're advocating to grow without any bound that I can see. I have some experience with trying to use a Datalog-like append-only model inside an SQL database, that's where these concerns come from.

As a general principle of rhetoric, if you seem to be saying something is easy or obvious but the reader considers it difficult or non-obvious, you lose something. Persuasion requires agreeing with the concern and then describing how you would dispel it. This sort of reasoning is why I tend never to claim anything is easy by my approach. I express doubt in an effort to hoard credibility with the sort of discerning audience I'm trying to attract.

We live in an era of absurdly fast machines with absurd amounts of storage. The industry has taught us to think everything is slow, in part I think so we stick with limited services on expensive hosted services.

Ahem.

Two responses:

• I’m describing a semantics. It’s a pretty novel and useful one. That comes at some cost in terms of runtime and storage. That may or may not work for various applications;
• I’m particularly interested in tools for individuals and small businesses. You could keep all your personal data, and the data for your carpet laying business with 10 employees, in a database system along the lines of what I describe. It would be “inefficient” in storage and computing time, but it would be respectful of what you really want to know and do as an individual or small business owner, around the data that are important to you.

I’m afraid I don’t understand the rhetoric question.

My point is I had some experience where what seemed like a small amount of data blew up once I added versions and tried to always append rather than mutate rows. On absurdly fast machines with absurd amounts of storage.

I suppose I'll wait until you have something to try out before finding out if it works for the sort of case I care about. So we can forget about rhetoric and wait for the code 🙂

Like I say, this would work for some applications. Amazon probably wouldn’t care to use this across all their data. The carpet laying business is a different story.

I’d love to see what your use case and look into why it broke. The most obvious way of implementing this, properly indexed, shouldn’t make data retrieval more than say twice as slow, I would think.

Yeah. Unfortunately this particular thing was in the context of a job and I don't have access to the code. It wasn't a lot of data and it was a simple CRUD webapp. But the database did need to be queried online. Performance was not an issue when we were mutating rows, but very quickly became an issue once we started appending rows. Perhaps I didn't know what to index. But that too is a problem your users will face, unless you somehow magically index the right things.

Perhaps the risk is that appending rather than mutating makes us more sensitive to indexing strategy. I can easily implement a database with mutation, but perhaps I need to learn something before I'm as effective at an append-only database.

Given my interest in smaller apps, I think we could happily have the database do its own indexing automatically. I’m kind of puzzled why this isn’t at least an option in major databases.

In case it’s not clear: the primary key on a table with a synthetic id column, an at column and a model_id column would be (model_id, id, at).

I think one difference with my experience may have been that I had to look up the most recent version id for rows. If you always know the precise model id that you need to look up, that may work well and my experience may not be relevant.

If you're looking up by a fixed model id, does that mean that any time one creates a new fork they need to copy all the rows from the past into the new model id? That would be a slow operation, right?

If you do it a hundred times, the database would grow by a factor of a hundred.

I suppose we could do something like what git does, let you check out multiple models. Checked out models would be fast to query but occupy linear space. Models that aren't checked out could be compressed to occupy sub-linear space.

You would probably want some forks to be copying, but I imagine mostly they would be lazy. So when you look for person with id 1, in model A, we would look for something like:

SELECT * FROM person WHERE (id, model) = (1, A).

If A was branched from another model, you would need to extend that with either a recursive CTE or you query for something like:

SELECT * FROM person WHERE id = 1 AND model_id IN (A, B, C)

assuming B, C are the antecedents of A in the branching hierarchy, and then you choose which model’s version of the record you should have based on the branch mode or whatever you call it.

To be clear: you could automatically add that logic behind the scenes. A developer using a Frest system would do the moral equivalent of:

SET model = A;

SELECT * FROM person WHERE id = 1;

(“moral equivalent” here because I’m very keen on making Datalog the query language of such systems…)

Yeah. That's the kind of thing that introduces some risk it'll get slow. So there's a trade-off here. You can complicate the query so performance becomes harder to reason about, or you can create a UX where people have to check out, and it'll make sense to people that those operations are slow. I think I'd lean towards the latter approach. I'd rather provide predictable performance and transparently expose slow operations to people.

Excel is also much slower than doing the same calculations in assembly language.

Branching and merging models is a frequent requirement in scientific computing. Most people still do it by hand, meaning badly. The only management solution I am aware of is git. You put all your code and all your data into git, if necessary using git-annex for big files, and then you have branching and merging. The UX is often a pain, in particular when your data is not in some text format.

So I agree that doing this properly promises to be a huge productivity gain. I just don't see how Frest is the solution. Sure, all data can be mapped onto a relational model. But to turn that into good UX, the mapping has to make sense to a user. For much of the data I deal with, that's not obviously possible. Though I agree that for my personal data it should be OK.

It remains to do it to find out. Excel and Access are the data manipulation models most amenable to understanding by non-programmers. I am fairly convinced that functional and relational models are the best for non-programmers to follow.

I believe that if you give the user tools to see the provenance of what they’re looking at — and this is an append-only first model, where most things have a complete history — then you can make this work.

It will certainly work for some folks, such as yourself. But I think the owner of the carpet laying business can make simple and effective use of this feature without having to do anything too confusing.

I am currently trying to sort out enough income that I can dedicate myself to building an app that will demonstrate these ideas. It would be a FileMaker/Access kind of app that integrates not just local tables, but remote information sources — a single Datalog query interface where you can join your email to the content of local files to your calendar to any and all online data sources.

Know an Angel investor who wants to back the idea? This would be a phenomenal application within which to deploy an AI…

I agree 100% that a good model of computation doesn't have to be for everyone. And a relational programming system without database cruft (SQL etc.) sounds like a good match for many use cases.