How we reduced the cost of building Twitter at Twitter-scale by 100x
Rama unifies computation and storage into a coherent model capable of building end-to-end backends at any scale in 100x less code than otherwise. Rama integrates and generalizes data ingestion, processing, indexing, and querying.
A summary from @Dustin Getz news.ycombinator.com/item?id=37137567
đ How we reduced the cost of building Twitter at Twitter-scale by 100x
Iâm going to cover a lot of ground in this post, so hereâs the TLDR: We built a Twitter-scale Mastodon instance from scratch in only 10k lines of code. This is 100x less code than the ~1M lines TwiâŠ
I have to admit, I'm confused by the presentation here. Maybe this is just personal, but even if there is something to be super excited about, it is hard for me to get excited. I'm all for database stuff. I know clojure so I understand where some of this is coming from. But the presentation just makes it so I have to dig and dig to try and understand what is going on.
So much cloak and dagger for backend infrastructure. I'd be intrigued to read their docs when it's more "open" -- /remindme august 22
I like that they are confronting the chronic impedance mismatch problem. But whatever new ideas they have are buried within a giant pile of custom tech they want you to adopt.
Bryan Cantrill is on it:
đŠ Bryan Cantrill on Twitter: You may not like it, but this is what 100X looks like
Oh well, in that case...
Edit: there are few people I trust as a technology-barameter. Bryan is one
This thread from one of the authors may be a better introduction than the article twitter.com/nathanmarz/status/1691509844828942336
đŠ Nathan Marz on Twitter: 1/ Announcing today our 100x programming platform called Rama and how we used it to reduce the cost of building Twitter at Twitter-scale by 100x https://t.co/GkV5MgWfmv
I just finished editing and published the talk from Rust East Coast. Perhaps some programming theory folks could find it interesting! cc Garth Goldwater
In a large Rust codebase, it can be difficult to find just the code you need to implement a feature or fix a bug. This talk will introduce Flowistry, a tool for visualizing data dependencies in Rust programs that can help programmers quickly sort out irrelevant code. I will explain what makes static analysis of dependencies a difficult problem, and I will describe the key insight in Flowistry that makes the problem tractable.
Presented by Will Crichton
Introduction: Situating Critical Code Studies in the Digital Humanities
Critical code studies is the application of the hermeneutics of the humanities to the interpretation of the extra-functional significance of computer source code. âExtraâ here does not mean âoutside ofâ or âapart fromâ but instead it refers to a significance that is âgrowing out ofâ an understanding of the functioning of the code.
This piece also links to a few other interesting resources on the subject, including, the initial manifesto for critical code studies.
I also liked this quote,
Still, reading code, even without interpreting its cultural significance, can be no easy task. Ask a professional programmer who inherits legacy code to maintain or, worse yet, to improve, and they will tell you about the dread of sorting out just-in-time code, minimally documented, written with hasty patches, full of compromises and workarounds. Even those who write their code in artistic projects can be shy about sharing their code out of embarrassment and self-consciousness. This shame is a product of the âencoded chauvinismâ of programming culture, one that can be fostered on the internet as much as it is in classrooms [ Marino 2020 , 148].
an absolute banger of a quote from the manifesto,
Currently, all of computer code lies before us with single quotation marks preceding its lines. While we examine programming architecture and admire modularity and efficiency, the study of computer code does not currently emphasize interpretation, the search for and production of meaning. Even when aesthetics intervene, they come in the form of calls for stylistic clarity for more efficient modification and development. This emphasis on functionality neglects the meaning that code bears for its human audience. For code, especially mid- to high-level languages, exists not solely for computers, which could operate on machine language (essentially, representations of electronic signals), but for programmers as well. Therefore, the computer may be one recipient of the code but there is also the programmer, other programmers, and at times even users who have access to its text.
Today's workshop on Unit with @Samuel TimbĂł is now up on YouTube: youtube.com/watch?v=lvvzolKHt2E
Thanks to everyone who attended, and to Sam for building and demoing such an impressive piece of tech!
I just want to say thanks to all for watching and participating! Kudos to Kartik Agaram for the curiosity and Ivan Reese for the leadership! I think we did cover a good amount of Unit basics, at least "enough to be dangerous"... I will tell you my (textual) programmer brain really knoted many times trying to implement some complex machinery in Unit, but when I "get it", it feels really good... it is like a puzzle... đ
I really appreciated some things about unit.land:
One of many things that I like: you're always building the code with a bunch of live example values flowing through it as you build. Like, the data is already present and flowing before the code is even plumbed together. This is why I'm so keen of visual programming â a spatial canvas makes it easy to trade off how much space is allocated to the code vs how much space is allocated to other helpful stuff. You can't easy change that balance in a traditional text buffer.
I thought this article on why some writers use an old word processor called WordStar was interesting. I wonder if there are any implications here for editing programs? Some of the ideas â like comments â we usually take for granted in programming. But maybe there are some other worth-while take-aways?
Let me speak generally for a moment. I've concluded that there are two basic metaphors for pre-computer writing. One is the long-hand manuscript page. The other is the typewritten page. Most word processors have decided to emulate the second â and, at first glance, that would seem to be the logical one to adopt. But, as a creative writer, I am convinced that the long-hand page is the better metaphor.
Consider: On a long-hand page, you can jump back and forth in your document with ease. You can put in bookmarks, either actual paper ones, or just fingers slipped into the middle of the manuscript stack. You can annotate the manuscript for yourself with comments like "Fix this!" or "Don't forget to check these facts" without there being any possibility of you missing them when you next work on the document. And you can mark a block, either by circling it with your pen, or by physically cutting it out, without necessarily having to do anything with it right away. The entire document is your workspace.
On a typewritten page, on the other hand, you are forced to deal with the next sequential character. Your thoughts are focussed serially on the typing of the document. If you're in the middle of a line halfway down page 7, your only easy option is to continue on that line. To go backwards to check something is difficult, to put in a comment that won't show when your document is read by somebody else is impossible, and so on. Typing is a top-down, linear process, not at all conducive to the intuitive, leaping-here-and-there kind of thought human beings are good at.
This makes it sound like someone made a deliberate choice to emulate typewriters rather than paper in computers. Computer text looks like typewriter text because it was influenced by the same (very powerful) force: linearity is easier to implement mechanically.
With that in mind, while Wordstar's block marking thing sounds cool, it sounds like a far cry from anything I would say resembles a long-hand page. I guess it doesn't take much when the competition is barely trying.
while Wordstar's block marking thing sounds cool, it sounds like a far cry from anything I would say resembles a long-hand page. I guess it doesn't take much when the competition is barely trying.
Yeah. Iâve been reading a WordStar manual to see if there is more to it than that, but Iâm not too far yet. So far, sounds a lot like a good text editor, but itâs hard to be fair from 40 years apart.
I used WordStar a bit on a CP/M machine in the 80s. It was very well designed given the limitations of those machines.
One thing Iâd like point out in Sawyerâs discussion is that WordStar has one of the best attributes any piece of software can have: it rewards mastery. One can argue that the keyboard shortcuts are harder to discover than, say, drop down menus actuated with a mouse, but once theyâre in your hands you can move faster than in any mouse-y interface.
my thoughts as a digital garden⊠kinopio.club/wordstar-cultural-aspects-9EAJDZCaN6dF96TNXD1bY
đ Wordstar Cultural Aspects â Kinopio
Kinopio is the thinking tool for building new ideas and solving hard problems. Create spaces to brainstorm, research, plan and take notes.
Seems a bit rosy. I use comments and outlines in gdocs to do the kind of thing they talk about and it works for me. Comments that don't show for others is a cool idea but also not exactly emulating traditional longhand (unless you use invisible ink?) And I'm pretty sure you can share a doc without comments, too, or just export to PDF.
I've never heard of or had experience with WordStar but I guess I feel modern tools are adaptable enough to cover these use cases. You might not feel your workflow is as privileged in the software, though, which I know matters.
Circling bits visually could be cool in code, though. A nice low-maintenance tool for highlighting potential areas of interest or problems.
Maybe a toggleable whiteboard layer over top of the whole file so you can draw arrows from A to B or sketch pictures in the margin. Would be difficult to align with stuff like tab sizes and fonts though.
I'm with Andrew. Text files are a 1-dimensional sequence of characters. We use them because they are easy for computers to use, not because they are easy for people to use. The fact that "code" is synonymous with "sequence of characters" is a problem. đ
WordStar has one of the best attributes any piece of software can have: it rewards mastery. One can argue that the keyboard shortcuts are harder to discover than, say, drop down menus actuated with a mouse, but once theyâre in your hands you can move faster than in any mouse-y interface.
This seems to be surprisingly hard to do with modern software. Part of it is variety: I have a lot of word processors I have immediate access to, and so I haven't mastered any of them. (Nor, I suspect, do they reward master much.)
I wonder, though, how much also has to do with our cognitive capacity. Modern software may have the same features, but unless I can get the features "in my hands" (mouse or keyboard or whatever I don't care, but it does need to be able to be muscle memory) the features may be about as distracting from creative thought as they are enabling .
If anyone wants to look at a WordStar manual, here's one: eaw.app/Downloads/Manuals/CPM/Wordstar_Training_Guide_2ed_Feb83.pdf
I wrote my Master's thesis using WordStar on an Epson PX/8, a portable CP/M machine that was sold off at 300 DEM in 1989 in Germany to students like me. Imagine: I could work on my text right in the university library, with all sources at hand! There was a learning curve to WordStar, but after a week I was as efficient as I ever was later with whatever other software.
Today, my main writing tools is Emacs. Its org-mode has a rich choice of informal markup, i.e. markup that is not interpreted by Emacs itself but still relatively easy to process semi-automatically via keyboard macros. That's great for the use cases described in this article. The tough part is being consistent in my own use of this markup. One day I'll write a linter - one day.
The other aspect that Emacs shares with WordStar is what Jack Rusher has pointed out: rewarding mastery. Much of today's software is designed for low engagement. Worse, engagement is punished by frequent redesigns.
Worse, engagement is punished by frequent redesigns.
Good point. Continuous deployment is detrimental to UX.
