Thank you, thank you. Hello. This will be eight or AT Protocol related, just so you know. The description does not make it sound that way, but I promise it will. For one thing, there's a leaflet uh link. So uh as Boris said, I have been I've been writing about this for a while. I've also been thinking about this topic for and and speaking and writing and and like prac practicing things around this for uh almost 20 years. Um so this is a um and that sounds probably pretty crazy because we haven't had generative software that long, but I'm gonna talk to you about architectural patterns today.
But I want to kind of set it up and and uh tell you about what I was thinking. So, you know, probably a lot of us started thinking about vibe coding before it was called vibe coding, you know, three more years ago. And the first thought I thought, you know, back in 2023 was like, you know, someday people will actually like allow agents to write code and they won't even review the code. That sounded crazy, you know. And then it's like, oh, next year, okay, a lot of people are gonna do this. And and now I'm thinking, of course, everyone's gonna do it.
Why is everyone gonna do it? Because people are lazy. Right? They are. They are lazy, agents are not lazy. And all of us with the best intentions, when we are stressed, we will do the easiest possible thing. So I started thinking about if we know they're gonna do it, how can we make it okay? And I should say it this way so it doesn't sound like I'm judging anyone. If I know I'm gonna do it, how can we make it okay? And not just for trivial vibe coded one-shot apps. That's a demo, that's a magic trick.
That doesn't matter in real life. It's fun, but it doesn't matter. And so the the thing that occurred to me is, and I was talking to a bunch of nerd friends, you know, old people like me about if I'm looking at something like uh a pure function written in Haskell. This isn't Haskell because I didn't want you to have to try to read that. Uh pure function written written in Haskell, so no no side effects, great type system, the language can ensure there are no side effects, you know what the input and the output types are, the names are great, etc.
I pretty much thought even three years ago, I can tell Chat GPT in the little chat GPT program to create that. I can just copy and paste it in, probably trust it, never have to verify it. We've all sort of had that intuition that small focused things are probably okay, and it's probably gonna do a good job, right? And so I started thinking about this like, well, what's the difference between that and a whole system? And really the whole system, it's it's about shapes. So I just described a bunch of things that were about shape, it was about size, it was about like the boundaries of the function, the type system, the names, all these things.
It's easy to replace something that is that small. It is also easy to generate something correct that small. So this led me on a whole adventure, but I'm gonna go back in time a little bit to 2014. Did any of you live through Heart bleed? Yeah. So if you were working at a company that had any production uh systems when Heartbleed happened, I was at Wonderlist in Berlin at the time I was CTO. I woke up one morning and uh my colleague James Duncan Davidson, who is the creator of Tomcat and Ant, like another old school guy, had sent me a message saying, we have to turn off the entire service.
That's how I woke up that morning. I was like, oh my god, what is happen what did we do wrong? We didn't do anything wrong. So Heartbleed was a bug in OpenSSL that it was just a mistake that allowed leakage of private data just to anyone on the internet that could access uh the service. And it was a disaster. It was such a disaster, so we had a microservices architecture, and we've been doing something called immutable infrastructure long before that was a thing that everyone did. It was before Docker was like in uh major circulation and usage, it was still pretty early.
Uh and so we had hundreds of services, which means we had hundreds of computers that had a bad version of Open SSL that were running in production with millions of active users. So what did we do? I mean that's uh that's a panic situation. I remember looking at the Amazon A AWS dashboard at all the things and just thinking, what are we gonna do? We were waiting for Amazon, they were not responding, they were very busy, as you might guess that day. And so we said, well, as soon as we have a good open SSL image, we're just gonna replace literally every server.
Hundreds of servers, 300-ish instances at the time. How did we do it? I went to a computer, I was afraid, I typed a command, and over the course of a couple of hours they all got replaced, nothing ever went down, and it just worked. Why? Because the system was built to s uh to be a massive system of small components, and it was always built with the idea that these components would be destroyed. That's the idea idea behind immutable infrastructure. You never change a server, you only modify or the only way you modify a server is by replacing it.
So if you need to upgrade something, you throw away the old thing, you replace it with a new thing. Kind of old hat and infrastructure now. If you're a DevOpsy kind of person, that's how you do it, probably. If not, apologies to you, you should get a new job. So it took us a day. It took most teams weeks or a week. There are still 250,000-ish servers on the internet that are suffering from this problem. Which means 250,000 compromised servers that you can get private data out of for whatever reason. Okay, so but this problem is sort of solved, right?
This is more than 10 years ago. Here's an imaginary scenario. Later on this year, the Lazarus group in North Korea infect some packages in NPM or whatever package manager. And the infection goes like a virus through the network and it starts infecting all the other packages. And at the end of the day, you wake up to the news that the entire JavaScript ecosystem is compromised. What do you do? You can't delete all the servers and restart them because NPM itself is compromised. Everything is compromised. You can't trust anything in the ecosystem. You have to actually move to a new language and a new ecosystem.
So let's do that in a day. Yeah? On a complicated uh application. Probably not, right? But that's the idea. You would need to delete everything and regenerate, regenerate it. It would be an absolute disaster. So this is the goal of the architecture that I'm putting forward here. And you probably can't read it because it's too small. These are command lines that say like Phoenix generate and regenerate, and the idea is just like replace all the implementations of everything that we have, say like from NPM, we're gonna go ahead and finally move to Rust like everybody else.
So one last story. Early in my career, this is over 20 years ago. Uh I was going, I was like moving up the career ladder. I was at a big public company, I was moving toward a CIO position uh uh in one of the sub, you know, the divisions of the company, and I was asked to do a new job. The new job was pretty surprising giving my career path. It was to be the lone developer that took over another system from a guy who was trolling the company. So this guy had written this really complex system in C that was talking to the Moby Tech's radio frequency network.
It was managing laptops in the field for service technicians. It was basically like a mess a bespoke message queuing system, like you'd do rabbitMQ today, he had built his own thing then. No one knew how this thing worked. Why? Because the only real artifact of the system was the code. It was a bunch of conversations that had happened in ugly conference rooms in this company over the years. It was all in production. This guy was the only person who now knew how any of it worked. Really knew how it worked. But he was basically holding us hostage as a company.
So I went from move towards CIO to go sit up all night in an office at in in the dark, like doing S trace on server processes, trying to figure out what this thing does. So this can happen today, and it could happen 20 years ago. You can be held hostage by your code base. So I learned something very early around that time in my career from uh guy named Kent Beck, who is the creator of extreme programming and a bunch of other stuff. And if you're my age in software development, he's probably one of your software development uh heroes.
Um if something hurts, do it more frequently. So with extreme programming, which was the precursor to agile, which ruined everything, and they should they never should have done the agile thing, but extreme programming. It was like merging is hard, you need to do continuous integration, deployment is hard, you know. We learned all these things, uh testing is hard, you should do it up front, so no code is written without a test. Code review is hard, just do pair programming constant, right? So if updating code is hard and making changes to code is hard, the Kent Beck version of that would be just do it constantly, like all day, maybe.
So how do you do that? That means that the code itself is not that important, sort of, right? If you're gonna be replacing it all the time, if you're gonna be throwing it away and replacing it, how do you do it? Uh and then another person who's not as smart as Kent once said the mutability of a system is enhanced by the immutability of its components. I said that, but you know, I I I like to quote myself because I think it's a pretty good one. So maybe in 20 years or so, someone will say, if you're me, you heard Chad say this, and it was meaningful to you.
So the weird thing about this is, you know, we're in this this weird era. How many of you are actually software developers, by the way? Okay, got enough of you that that some of you will feel sad about this sort of, you know. Is the craft going away? I've been posting a bunch of stuff on that leaflet, and people get mad. Every time I post something, there's at least someone who is angry and like, you know, you're wrong, idiot, you are influenced by marketing. That was yesterday's post, actually. Um, but the craft doesn't disappear, it just moves.
So, you know, from like doing test driven development, you need to be thinking about how you evaluate the code from a business perspective and from a like architectural perspective in terms of what the technical requirements are of the code. Um, you know, you'll be thinking about things like what are the boundaries and what is the actual architecture that can work in a system that's constantly replacing itself. How can that possibly work? And so I'm gonna dig into that kind of stuff. One note here though, is there is still very much a craft or an engineering discipline or whatever sort of metaphor you want to use, art.
You know, I'm a musician, so maybe uh there's a craft or there's an art to it. And what I'm seeing now, and the fear that a lot of people have in the software industry is that the senior people, and senior doesn't have to mean old or even experienced, but somehow the people who have gained wisdom in this field somehow, they are moving farther and farther away from the newbies because everything moves so quickly, and the the mistakes you make have such great and instant ramifications, and the good choices you make have such major ramifications that this AI thing amplifies the effect of this experience.
So, Phoenix architecture, what is it? It's a name that I I let the AI choose, honestly. This is a thing I do because naming is one of the hardest things, right? So that's why we have LLMs. You don't ever have to name anything again. Just make sure it's not offensive in any language if you can, but you can ask the LLM that too. It's an architectural idea, a bunch of architectural principles. Uh, as I said, they are extracted from my decades of thinking about this problem and trying to figure out how we can create systems that can survive.
So if you if you Google for me, if anyone uses Google anymore and put the word legacy, you will find me talking about this back in 2009, and some of the things I'm gonna say are the same today. Because I realized that making systems that can survive, the principles have been the same for a long time. Um it's also a reference implementation of this architecture. And I'm gonna show you two things very briefly uh on the reference implementation, and you can find it on Tangled, or you can find it on the copy on GitHub. But obviously you're all gonna use tangled, right.
So I'm starting to think about software not as a thing where you you go into an editor, you know, that's always been like a short version of what the AI thing would be, these copilot things in an IDE that's gone already. But rather a pipeline. So we go from specifications to something else to something else to something else that ends up generating code. And and the something else is here, you won't be able to read this, but you can just kind of see columns here. Imagine you've got a bunch of markdown specifications. Those are just things where you just type stuff.
Maybe you're also in a chat program, like uh freak. Ask me about freak, F R E E Q. Um that will be extracted clauses. So you can use LLMs for this. It's not quite you know perfect, but my reference implementation does some hacks. You extract clauses, which are like different ways of saying the same thing, you can kind of deduplicate them into this text chunks. From that, you can canonicalize and deduplicate actual requirements. From that, you can expand them into various things that you want to evaluate. They could be invariance, they could be various other constraints.
They might even be things like this component must be fast, which turns into this component must respond within 50 milliseconds, so that becomes a constraint. And from there, you go into the implementation, which I'm gonna talk about in a second, has two phases sort of necessarily. The cool thing about this is uh if you if you create a system like this where you're actually thinking of that end piece, the code as a compiled artifact, think of it as a binary, like TypeScript or whatever you're generating. It's like binary code, you wouldn't edit it. It also means that you can trace the provenance from the markdown file from a piece of text in the markdown file through this graph of requirements all the way through to the generated thing at the end.
So just like you were doing a C program or something where you're compiling from C to an object file to blah blah blah, the TypeScript is just in that dependent path. And so when you set up a system like this, you can do two things. One is you can regenerate just the path down to the component at the end that you want to generate. The other is if someone does do the wrong thing, so I say, and edit the code themselves as a human, you can see where it drifts. And you can see, well, well, the drift is here and it actually affects this spec.
And I could even like automatically uh analyze the code and see it's changed the spec in this certain way. Is that actually the spec now? Is that what you meant? And commit it as the real intent of the code. Because intent is what matters more than anything else now, intent plus architecture. Let me put this back in full screen. Or not. This is the fun of creating your own presentation software. So when I was implementing this, I realized when I got to a certain point, I'm gonna generate some code. What the hell am I going to generate?
Like, is it a React application? Is it a Next.js application? Is it a Rust thing? The answer is I don't know. And not only do I not know, I don't want to have to for a given application decide those things. Because I want to be able to move from, let's say, JavaScript to Rust when the Lazarus group takes over NPM. They're not going to, but if they did, I want that flexibility. And the real life story behind that is at Wonderlist again, we launched Wonderless 3, which was our huge rewrite that finally made the sync work and everyone was po everyone was happy with it.
We launched it with Ruby on Rails as the back end. Lovely for humans, not great for compute, uh, very expensive to run. But the way we had architected this back in 2013-14, allowed us, in the for the same reasons that we could fix the heart bleed problem, allowed us to rewrite each service in another language within three months. And we went from uh, I'll just say we saved 70% of our compute costs by rewriting services without affecting the system because we were just replacing the components. So in a system like this, if we're building this for agents, if we're building this for a constantly regenerating world, we need to compile not to an implementation, but we need to compile to an architecture that has these properties of of components with boundaries that are just like that Haskell thing I was talking about earlier that can be replaced at will because the boundaries are clear enough.
So the idea is you have some intent, maybe it comes from a conversation. Usually they come from conversation, maybe documents, lots of conversations. The compent the intent is then compiled into some decisions, not into code necessarily. The decisions then generate code, and then they go through an evaluation loop where you see, well, is this code actually doing the thing that it's required to do based on the specs, including all those invariants and and constraints and stuff I was talking about earlier. That could even be stuff that's coming from observability systems. So I think software development in in the near future is a closed loop from production where you're actually pulling in what's happening in production, it's informing and saying, well, this thing has now drifted from the requirements.
I think we should regenerate these two components because they're not doing the thing that the business says they need to do, right? So then you record the provenance, which means the chain of the human or the age human and humans' agents, and all of the different specs and everything else that led to the generation, capture it forever cryptographically. Then you deploy it within the boundaries, same way I talked about with Wonderlist. So you're deploying these nice tight uh controlled components. And then I won't talk about this now, but if you read the blog, you will compact because we are creating tons and tons of garbage, those of us who are making uh stuff with LLMs now.
I accidentally created two web interfaces for my IRC network, and it took me a while to delete them because I couldn't remember which one was the right one. So there you go. So I should mention what is an evaluation, because this is an overloaded term now. I kinda got into it. It's all of these different things, you know, the the requirements that could be just like it must act like this, you know, what we're used to with specs and user stories, but it's also it shouldn't use this much memory or it has to be this fast, or it has to be, you know, deployable on this thing or whatever.
So evaluations are the thing that are about the system and not about the code. It's not unit tests, because unit tests are tied to actual implementation. So that's not the thing that you want to rely on. That's also a thing that they can generate better than we can, and we should let them. So it might be has to have 100% test coverage and they always have to pass that sort of stuff. That could be one. And then here's the sort of secret. In the flow I showed you earlier, where you have specs going to canonical uh requirements, et cetera, et cetera.
There are two things in the there are two like columns and implementation. Before you get to the actual implementation, I'm calling this implementation units. Because this is actually the decisions about what components should exist and what their behaviors and boundaries and interfaces need to be. This should not be language or runtime dependent. This is the promise of a thing that will be implemented in some language and deployed in some runtime that meets the requirements of the specifications. And this is a really powerful thing because it means you can do things like say, well, let's just do a pass this afternoon and see if we can make this thing 50% faster.
Okay, good. Friday, deploy it, you know. That's where we want to get with this stuff. So implementation units are sort of like the semi-abstract version of the the thing that you want to test, and they're what the actual evaluations tie to. And the implementations can have unit tests and all the other stuff that you're used to in your language. But we are actually past a point now of I think, sorry, language advocacy. And I say that as someone who like, you know, I was like, I started the Ruby nonprofit and the RailsConf and RubyConf and wrote the RubyGems thing and spoke at Ruby conferences and wrote books about Ruby and I was all Ruby Ruby Ruby 20 something years ago.
And uh I'm going to RubyConf to receive an award this year and tell them you probably ought to not use Ruby anymore. It's not a good idea. Because who's Ruby for? People. Ruby is for people. These programs are for machines. And what matters now is that they're fast, correct, you can generate them with LLMs or whatever the technology is gonna be, they're cheap to run. That's what matters. Doesn't matter what language it is anymore, so let go of that, except for your hobby time. Write it in your hobby time, fine, fun. Otherwise, it's irresponsible. And that was a, yeah, just throwing some bombs at you here.
All right. And so you know the code doesn't matter anymore. We have to let go of that. We have a code fetish. I've been saying that for decades, literally, to developers at conferences. We have a code fetish, it's not about the code. Now it's about these evaluations, really, and the fact that things need to exist. So sort of backing up a little bit, think about like I I went through the whole microservices movement, if you can call it a craze hype, whatever, and I was definitely one of them. Uh, and I'm uh I'm friends with Fred George who created that term and probably regrets it now.
People hate microservices. Now there's been a movement to move back to monoliths. That's also really stupid and misguided. The reason they hate microservices is like imagine you you spend six months like breaking apart apart this monolith into a bunch of microservices because your manager said we're doing microservices now, and then you finally remove the thing and you realize, oh crap, there are all these dependencies we didn't really understand that are like maybe they're temporal or they're data related or whatever, you didn't accomplish anything by doing this. Maybe some sort of like production runtime improvement of not having you know services get on each other's uh um you know CPU cycles and slow the whole system down or whatever.
Not good. So this is sort of a necessary thing to think about. Like it's not just small, but it's about boundaries and a replaceability. That's what matters. And and I say this as a person who did a talk called tiny 10 years ago at a conference, thinking I think it's just about making things small, guys, but it was not. Now what I'm realizing, and this is me also shelling shilling my project, I must admit, but uh really it's about the conversation. So, you know, I was thinking about like how do you apply all this generative AI stuff?
It's it's fine for one-shotting apps, and you know, like I made this presentation with it, I don't have a slide deck, I just have talking to an uh LLM having it generate things. Um what really matters though uh on big systems is context. Like how do you get the intent? How do you go to a 20-year-old system and find out what the intent is behind the code? You probably do not, actually, right? Because no one captures it, there's just no system for that. There never has been, except in like really really bureaucratic places, but then it was malicious uh uh compliance and you're gonna end up a crap anyway.
So, what if we could create an environment where the conversation is always captured around how the requirements were created and specified, tied to a human in a way that is archived and will never be lost when you move from Slack to Teams or something. Uh and every commit and every piece of code is tied to that conversation and the people that did it, so you get a sense of why. I think this is the only responsible way to do software development next year. And to do that, it means you may not touch code as a human.
You can view it, like you shouldn't even be able to you can use VI, it's okay. If you use VI, you can change it, but you know, like don't use VS code and edit your code. It's wrong. Because if you as a human do it, you can't guarantee all this stuff. You won't guarantee being able to capture all this provenance. And bots are way better at discipline than we are. So I'm gonna go kind of fast now. I talk about regenerating code, and I talk about like systems, like in in my perfect world, a system in production maybe doesn't have any of the same code it had last month.
And that's always true. It's just always rolling. And probably I will think that's quaint in a year and say, really, I meant five minutes. Sorry guys, you know. That's sort of insane though, because just because we can change it doesn't mean we should. And there are layers. So uh Stuart Brand has this concept of pace layers, which is really just about like layers of uh the ability to change and where things need to move more slowly and where they can move more quickly. An example of a paste layer that should move slowly is like a protocol, you know, HTTP, let's not change that daily, but also a user interface.
You can't just have the user interface change constantly because your users will be confused. Um I think you know Paul might have mentioned that earlier in their presentation. The other thing that we need to worry about is hidden state, and I sort of got to that with the with the uh deletion test idea of you know doing this big microservices thing and then you replace the monolith with microservices and nothing works. There's a bunch of that in the world. And so you to do an architecture like this, and this is gonna be different for every application.
This is a very serious thing, you need to be thinking hard about service boundaries, data boundaries, et cetera, even temporal uh coupling. The end sort of goal here, and really something that's always been true, is if we're on a team together, what we are building is not the code. We're building a system. And the code is not only not the asset, it is a liability. It always is because every line of code you ever wrote is legacy code that the moment you wrote it, right? Legacy code is bad, we all know that, right? So the system is the asset, code is a liability.
So if you think about all this stuff I was just saying, you think about anything in your current systems, like pick one component. Could you replace it tomorrow? Could you just have Claude redo it? If not, you probably have a refactoring job, a system-wide refactoring job ahead of you. And I will assert you already have that refactoring job ahead of you, you just didn't know it yet because things are moving a thousand times faster than they used to, or maybe faster than that. I don't know what the right number it is. So the point is that AI didn't actually create this problem.
And I think that's true of a lot of stuff we're seeing now, both in terms of the problems we see from AI and the solutions. It's like a real pattern I'm seeing. I I'm also a VC, sorry. Uh I I heard some shade to VCs on the last talk, and I agree with it, honestly. I'm a VC who's also a programmer. I talk to a lot of people, and a lot of things happening now that I think are really powerful are like AI is creating a problem because things move so fast, so much content's getting created, et cetera, et cetera.
So we're gonna use AI to fix the problem. And that's kind of where I am right now. But it's the same thing we always had as just going a thousand miles per hour. So all the stuff I was talking about, how do we make it more concrete? Well, I told you I have a reference implementation of the Phoenix architecture. I linked it earlier. But I also told you the conversation is the commit. And the things I was saying, I hope they were registering like, well, I can't use Slack as my system of record, or I can't use Teams or whatever, or Discord, right?
I need something with like durability that I can host myself that I can trust that no one else can read unless I want them to. Uh I want cryptographic identities. I want provenance that way. I want my agents to be spawned by me and my key, and I want them to have keys, and I want us to maybe even have reputation and durability. And so I decided I would just build something that does this. It started as a hack one day, and here's where it's an app protocol thing. I was like, what if we could sign into IRC, if you know what that is, internet relay chat for the the young people with your Bluesky handle.
And it just kind of got out of control. So now what I'm building is a substrate with all of these properties, agent provenance, et cetera, et cetera. There's a lot to it. Um, but the end goal is the substrate for agent-driven development uh with Phoenix. And so um, please join freak.at. Please contribute stuff. Tell me what's stupid, what's not great. It it kind of looks like there's iOS, Android, Windows Web Tui Rust SDK bot SDK great docs it's ready for you but I have not launched it and don't tell anyone about it but please join because it's just for nerds right now but I think this is how things are gonna work in the future and I guess I'm out of time so thank you very much for your attention thank you very much Chad um accidentally unencrypted IRC client with atproto accounts amazing thank you very much