You are viewing archived messages.
Go here to search the history.

Mariano Guerra 2023-08-15 19:58:26

How we reduced the cost of building Twitter at Twitter-scale by 100x

Rama unifies computation and storage into a coherent model capable of building end-to-end backends at any scale in 100x less code than otherwise. Rama integrates and generalizes data ingestion, processing, indexing, and querying.

blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x

A summary from @Dustin Getz news.ycombinator.com/item?id=37137567

📝 How we reduced the cost of building Twitter at Twitter-scale by 100x

I’m going to cover a lot of ground in this post, so here’s the TLDR: We built a Twitter-scale Mastodon instance from scratch in only 10k lines of code. This is 100x less code than the ~1M lines Twi…

Jimmy Miller 2023-08-15 21:09:37

I have to admit, I'm confused by the presentation here. Maybe this is just personal, but even if there is something to be super excited about, it is hard for me to get excited. I'm all for database stuff. I know clojure so I understand where some of this is coming from. But the presentation just makes it so I have to dig and dig to try and understand what is going on.

Walker Griggs 2023-08-15 23:09:14

So much cloak and dagger for backend infrastructure. I'd be intrigued to read their docs when it's more "open" -- /remindme august 22

Jonathan Edwards 2023-08-15 23:13:12

I like that they are confronting the chronic impedance mismatch problem. But whatever new ideas they have are buried within a giant pile of custom tech they want you to adopt.

Jonathan Edwards 2023-08-15 23:18:00

🐦 Bryan Cantrill on Twitter: You may not like it, but this is what 100X looks like

Tweet Thumbnail

Walker Griggs 2023-08-15 23:19:37

Oh well, in that case...

Edit: there are few people I trust as a technology-barameter. Bryan is one

Mariano Guerra 2023-08-16 08:15:27

This thread from one of the authors may be a better introduction than the article twitter.com/nathanmarz/status/1691509844828942336

🐦 Nathan Marz on Twitter: 1/ Announcing today our 100x programming platform called Rama and how we used it to reduce the cost of building Twitter at Twitter-scale by 100x https://t.co/GkV5MgWfmv

Cole Lawrence 2023-08-15 23:09:30

I just finished editing and published the talk from Rust East Coast. Perhaps some programming theory folks could find it interesting! cc Garth Goldwater

In a large Rust codebase, it can be difficult to find just the code you need to implement a feature or fix a bug. This talk will introduce Flowistry, a tool for visualizing data dependencies in Rust programs that can help programmers quickly sort out irrelevant code. I will explain what makes static analysis of dependencies a difficult problem, and I will describe the key insight in Flowistry that makes the problem tractable.

Presented by Will Crichton

youtube.com/watch?v=aYmuMlzvjvc

Eli Mellen 2023-08-16 11:29:51

Introduction: Situating Critical Code Studies in the Digital Humanities

Critical code studies is the application of the hermeneutics of the humanities to the interpretation of the extra-functional significance of computer source code. “Extra” here does not mean “outside of” or “apart from” but instead it refers to a significance that is “growing out of” an understanding of the functioning of the code.

This piece also links to a few other interesting resources on the subject, including, the initial manifesto for critical code studies.

I also liked this quote,

Still, reading code, even without interpreting its cultural significance, can be no easy task. Ask a professional programmer who inherits legacy code to maintain or, worse yet, to improve, and they will tell you about the dread of sorting out just-in-time code, minimally documented, written with hasty patches, full of compromises and workarounds. Even those who write their code in artistic projects can be shy about sharing their code out of embarrassment and self-consciousness. This shame is a product of the “encoded chauvinism” of programming culture, one that can be fostered on the internet as much as it is in classrooms [ Marino 2020 , 148].

Eli Mellen 2023-08-16 11:34:11

an absolute banger of a quote from the manifesto,

Currently, all of computer code lies before us with single quotation marks preceding its lines. While we examine programming architecture and admire modularity and efficiency, the study of computer code does not currently emphasize interpretation, the search for and production of meaning. Even when aesthetics intervene, they come in the form of calls for stylistic clarity for more efficient modification and development. This emphasis on functionality neglects the meaning that code bears for its human audience. For code, especially mid- to high-level languages, exists not solely for computers, which could operate on machine language (essentially, representations of electronic signals), but for programmers as well. Therefore, the computer may be one recipient of the code but there is also the programmer, other programmers, and at times even users who have access to its text.

Eli Mellen 2023-08-16 17:43:59
Ivan Reese 2023-08-16 21:49:04

Today's workshop on Unit with @Samuel Timbó is now up on YouTube: youtube.com/watch?v=lvvzolKHt2E

Thanks to everyone who attended, and to Sam for building and demoing such an impressive piece of tech!

Samuel Timbó 2023-08-17 00:05:39

I just want to say thanks to all for watching and participating! Kudos to Kartik Agaram for the curiosity and Ivan Reese for the leadership! I think we did cover a good amount of Unit basics, at least "enough to be dangerous"... I will tell you my (textual) programmer brain really knoted many times trying to implement some complex machinery in Unit, but when I "get it", it feels really good... it is like a puzzle... 😉

Kartik Agaram 2023-08-17 00:27:25

I really appreciated some things about unit.land:

  • It doesn't perpetuate the false dichotomy between inside and outside, interface and implementation. The code for a product is just another view on the product, and the interface for browsing the code is designed like a product. That duality and the easy way to bounce between the two promises lots of tight feedback loops in many dimensions. Other projects like Excel share this property, often by dumbing down the internals. Excel focuses on data to the exclusion of code, etc.
  • The code feels like an organism, alive. I was reminded of experiences playing StarCraft where you lay out a few things and then can often watch how things play out once the battle starts. The surface the code lies on feels alive like an eco-system, particularly when you take the hierarchical structure into account. It feels like a jungle you could immerse yourself in. And I really like the idea that giving it detailed simulation of physics leads to an aesthetic environment.
  • You can see values flowing through the code. Many other programming models struggle to integrate the two. Showing example values next to code feels like a hack compared to this. The way you pause computations feels organic. Spreadsheets hide the dataflow which hinders debugging.
  • Mobile friendly. I almost missed this. This is potentially huge. There's an almost 15-year old open question of how people might develop on small screens. This might be the answer.
Ivan Reese 2023-08-17 01:09:14

One of many things that I like: you're always building the code with a bunch of live example values flowing through it as you build. Like, the data is already present and flowing before the code is even plumbed together. This is why I'm so keen of visual programming — a spatial canvas makes it easy to trade off how much space is allocated to the code vs how much space is allocated to other helpful stuff. You can't easy change that balance in a traditional text buffer.

Ivan Reese 2023-08-17 02:15:09

(Meta: Also shared this on HN, Lobsters, and social media.)

David Alan Hjelle 2023-08-20 02:27:48

I thought this article on why some writers use an old word processor called WordStar was interesting. I wonder if there are any implications here for editing programs? Some of the ideas — like comments — we usually take for granted in programming. But maybe there are some other worth-while take-aways?

Let me speak generally for a moment. I've concluded that there are two basic metaphors for pre-computer writing. One is the long-hand manuscript page. The other is the typewritten page. Most word processors have decided to emulate the second — and, at first glance, that would seem to be the logical one to adopt. But, as a creative writer, I am convinced that the long-hand page is the better metaphor.

Consider: On a long-hand page, you can jump back and forth in your document with ease. You can put in bookmarks, either actual paper ones, or just fingers slipped into the middle of the manuscript stack. You can annotate the manuscript for yourself with comments like "Fix this!" or "Don't forget to check these facts" without there being any possibility of you missing them when you next work on the document. And you can mark a block, either by circling it with your pen, or by physically cutting it out, without necessarily having to do anything with it right away. The entire document is your workspace.

On a typewritten page, on the other hand, you are forced to deal with the next sequential character. Your thoughts are focussed serially on the typing of the document. If you're in the middle of a line halfway down page 7, your only easy option is to continue on that line. To go backwards to check something is difficult, to put in a comment that won't show when your document is read by somebody else is impossible, and so on. Typing is a top-down, linear process, not at all conducive to the intuitive, leaping-here-and-there kind of thought human beings are good at.

Andrew F 2023-08-20 02:45:02

This makes it sound like someone made a deliberate choice to emulate typewriters rather than paper in computers. Computer text looks like typewriter text because it was influenced by the same (very powerful) force: linearity is easier to implement mechanically.

With that in mind, while Wordstar's block marking thing sounds cool, it sounds like a far cry from anything I would say resembles a long-hand page. I guess it doesn't take much when the competition is barely trying.

David Alan Hjelle 2023-08-20 11:23:46

while Wordstar's block marking thing sounds cool, it sounds like a far cry from anything I would say resembles a long-hand page. I guess it doesn't take much when the competition is barely trying.

Yeah. I’ve been reading a WordStar manual to see if there is more to it than that, but I’m not too far yet. So far, sounds a lot like a good text editor, but it’s hard to be fair from 40 years apart.

Jack Rusher 2023-08-20 11:39:58

I used WordStar a bit on a CP/M machine in the 80s. It was very well designed given the limitations of those machines.

One thing I’d like point out in Sawyer’s discussion is that WordStar has one of the best attributes any piece of software can have: it rewards mastery. One can argue that the keyboard shortcuts are harder to discover than, say, drop down menus actuated with a mouse, but once they’re in your hands you can move faster than in any mouse-y interface.

Paul Tarvydas 2023-08-20 12:29:13

📝 Wordstar Cultural Aspects – Kinopio

Kinopio is the thinking tool for building new ideas and solving hard problems. Create spaces to brainstorm, research, plan and take notes.

Grant Forrest 2023-08-20 14:25:27

Seems a bit rosy. I use comments and outlines in gdocs to do the kind of thing they talk about and it works for me. Comments that don't show for others is a cool idea but also not exactly emulating traditional longhand (unless you use invisible ink?) And I'm pretty sure you can share a doc without comments, too, or just export to PDF.

I've never heard of or had experience with WordStar but I guess I feel modern tools are adaptable enough to cover these use cases. You might not feel your workflow is as privileged in the software, though, which I know matters.

Grant Forrest 2023-08-20 14:26:33

Circling bits visually could be cool in code, though. A nice low-maintenance tool for highlighting potential areas of interest or problems.

Grant Forrest 2023-08-20 14:27:59

Maybe a toggleable whiteboard layer over top of the whole file so you can draw arrows from A to B or sketch pictures in the margin. Would be difficult to align with stuff like tab sizes and fonts though.

Jason Morris 2023-08-20 20:58:48

I'm with Andrew. Text files are a 1-dimensional sequence of characters. We use them because they are easy for computers to use, not because they are easy for people to use. The fact that "code" is synonymous with "sequence of characters" is a problem. 🙃

David Alan Hjelle 2023-08-20 21:00:36

WordStar has one of the best attributes any piece of software can have: it rewards mastery. One can argue that the keyboard shortcuts are harder to discover than, say, drop down menus actuated with a mouse, but once they’re in your hands you can move faster than in any mouse-y interface.

This seems to be surprisingly hard to do with modern software. Part of it is variety: I have a lot of word processors I have immediate access to, and so I haven't mastered any of them. (Nor, I suspect, do they reward master much.)

I wonder, though, how much also has to do with our cognitive capacity. Modern software may have the same features, but unless I can get the features "in my hands" (mouse or keyboard or whatever I don't care, but it does need to be able to be muscle memory) the features may be about as distracting from creative thought as they are enabling .

David Alan Hjelle 2023-08-20 21:03:32

If anyone wants to look at a WordStar manual, here's one: eaw.app/Downloads/Manuals/CPM/Wordstar_Training_Guide_2ed_Feb83.pdf

Konrad Hinsen 2023-08-21 06:46:49

I wrote my Master's thesis using WordStar on an Epson PX/8, a portable CP/M machine that was sold off at 300 DEM in 1989 in Germany to students like me. Imagine: I could work on my text right in the university library, with all sources at hand! There was a learning curve to WordStar, but after a week I was as efficient as I ever was later with whatever other software.

Today, my main writing tools is Emacs. Its org-mode has a rich choice of informal markup, i.e. markup that is not interpreted by Emacs itself but still relatively easy to process semi-automatically via keyboard macros. That's great for the use cases described in this article. The tough part is being consistent in my own use of this markup. One day I'll write a linter - one day.

Konrad Hinsen 2023-08-21 06:54:40

The other aspect that Emacs shares with WordStar is what Jack Rusher has pointed out: rewarding mastery. Much of today's software is designed for low engagement. Worse, engagement is punished by frequent redesigns.

Paul Tarvydas 2023-08-21 08:03:03

Worse, engagement is punished by frequent redesigns.

Good point. Continuous deployment is detrimental to UX.