All right, I'm gonna go real fast because we've got a lot to get through. But I'm Jim. I run the platform team at Bluesky. Platform team is the uh team that runs our infrastructure, our data centers, cloud stuff, write a lot of backend code as well. And yeah, we have a lot to get through. So I'm gonna go super fast, but hit me up after if you have any questions or want to talk more. Who am I? I'm Jim. I live in Boston. I run the platform team as I said, been at Bluesky for about a year.
And here I'm here today to share some information with you on how we do stuff. Our atmospheric systems, our app view. I want to talk about what's going well, what could be improved, and maybe give some recommendations or at least like food for thought for you as well. All right, let's get into it. Who's this for? So there's a lot of people here who are in the weeds of atproto. A lot of people have different needs and wants out of atproto. And so this is really a talk for people who want to achieve high scale, such as running a big whole world app view, such as the Bluesky app View, right?
Another uh persona might be running a large fleet of PDSs. Uh Euros guys coming online, that's awesome. Uh BlackSky, like there's so much movement on this, and it's really cool. And I want to talk about uh well, okay. So not all projects are in that this category, totally rad. It's super awesome. Um yeah, I'll tell you a little bit about we what we do. This is just like fact dump, so we're just gonna get through it real quick. PDS fleet. We run about 110, wow, that's small. Run about 110 rented bare metal cloud hosts in uh US East and US West.
Um they range from 16 to 64 cores depending on what year we spawn them up. Uh they have 256 gigs of RAM, they have about a gig up, a gig down, and they have various disk configs. This is kind of fun and goofy. Uh learn about new disk configs every now and then. Uh most of them are XFS set up in a RAID one. Uh one of them I learned was running ZFS. Uh that's a little nifty experiment. It's been live for a long time. That's cool. Uh we also run one uh against our will on Ceph.
Um so that's kind of cool. Uh we had an emergency situation where we had to lift and shift a PDS, and all we had was a Ceph array in our data centers. Um that was uh right after Christmas. Uh they cost about 600 bucks a month each on OVH and I3D are the two suppliers that we use there. Uh and I'd say, you know, one thing on this, if you are looking to set up a big fleet of PDSs, we are way over provisioned on these things. So uh here's a Btop, and that is not showing up at all, but you can see we're using somewhere around like 5% CPU, uh, 5% of the RAM, so that's like 12 and a half gigs.
Uh it's doing uh relatively little network I.O. as well, even though uh you do kind of want to have at least a little bit of a beefy uh setup there. Um is like disk storage. We're kind of coming up on like actually starting to fill up disks, and we're gonna have to expand and stuff. Um so you can do a lot with a little with the PDS. Um I'll also say uh we have about a half million users per host. I'll say also that was my production PDS, so like that's Dapperling. There's a half million users running on that box.
Uh there's no special sauce. We're just running the open source code. Um there's nothing behind the scenes, there's no like rate limits bypass tokens or anything. It's just the code. Uh and yeah, we do I'll put a note on that. We have our own auth server. Um, but our config looks like this. We run an HA proxy on each one. There's 16 PDS containers. Each one of them has uh uh sorry, each uh user has a SQLite and we back those SQLite up with our clone once a day, and we do a light stream for live.
Uh we have some Redis, some Datadog for monitoring, Tail Scale for network auth. Uh it's all fully automated, zero touch provisioning. So like you just run one Ansible command, boom, we got a new one. It's really fast as a stand-up new ones. Second uh major topic is our pops. Uh pop is a point of presence, and it's basically like a uh it's a co-location center, it's a small data center. We run our own hardware that we own. We bought it. We uh we run the relay in our data centers, we run the app view, primarily powered by ScyllaDB in the data centers.
We run Discover in the data centers. There's a few other things, our search clusters in there. Um yeah. There's two of them, one in California, one in Ashburn, Virginia, and there's about 80 very large servers in each that we own uh and operate. We have super duper fast networks and uh it's easy to add more. Super fast disks and a lot of them. It's really high bandwidth, and we have active active everything. So there's like two ISPs, there's two copies of pretty much everything, and uh yeah, we want that high degree of redundancy so you can provide really solid service.
Here's some pictures. Uh this is the posting factory. You can see me and Austin, Austin's over there uh down on the back. My friend Patrick's actually behind Austin. Uh sorry, Patrick. Um but yeah, this is one of them. This is what a data center looks like. Uh so you get a cage that's like your full suite, and in it, you put a bunch of racks. Here's one of the racks within the rack, you have a couple of switches, and then you have a bunch of compute servers. Um, and then you have two of those, right? So two of everything.
Um it's kind of what it looks like. It's pretty bog standard. Uh next is our AWS count. I'm not gonna talk too too much about AWS, but it's where we run a bunch of like singleton stuff. Uh that's you know, lots of stuff in there, super important, some of it's kinda chill. PLC is like obviously really important. The main BSky.app website, like the HTML, the CSS, the JavaScript, the assets are served out of there, a few other things, and we have a ton of Postgres in there. Uh Postgres is really hard to run, it's really annoying.
RDS is great, it's very expensive. Um what's going well? Pops are goaded. Pops are the goat. They're super duper cheap and they're extremely high performance. Uh the PDS fleet is working quite well, actually. Um it's really easy to add more servers, so as we're growing, we can just chuck uh new new servers up. Uh they're very reasonably priced. Uh AWS is AWS, it's fine. Um yeah, RDS is rule, it's so good, it's worth every penny. Renting GPUs is quite convenient. We do run a bunch of GPUs up there for various things. Um besides that, it is very, very expensive.
Uh do some very rough napkin math, and I'm really not gonna get into this too too much, but um it costs us probably about 800 grand a year to run the pops, um, amortizing for uh depreciation of the assets over four years. Uh it's the roughly equivalent AWS install is literally impossible because our like our total switching capacity in the pops is just like shocking. Like it's crazy. Uh and you couldn't do our AWS setup, you couldn't do our pop setup in AWS, but if you did, it'd probably be about 10x that. So about eight million a year.
Uh and that's with like heavily negotiated long-term reservations. Probably about 14 million if you're doing on demand. Heavy asterisk, I vibe coded all that. Um so um yeah, vibe finance. Um that being said, the pops are an absolute shitload of work. It requires deep expertise. Austin has gone like absolutely crazy on uh trying to do our reprovisioning of all this stuff and make it sane and make it easy to work with. It has really slow iteration cycles. It once you're getting new hardware, it takes a while to get it online unless you have excellent operational practices.
Again, kudos Austin. Um RAM and storage also is up and to the right, unfortunately. And so we bought a bunch of stuff uh thanks to Jazz like here ish. Uh yeah, about here. So uh jazz is yeah, jazz is the goat. Um so what's work what's next? Uh and I'm running out of time, so I'm gonna go fast. Uh pops uh make them easier to operate, as I said, Austin's been doing Yeoman's work on this. Uh increase our compute density as well. Um previously we basically were assigning one service to a box, and like oftentimes the service would need less than one percent of one of the CPUs, and each one of them has like 256.
And so um I'm gonna say the evil Kubernetes word. Um yeah, trying to improve our density there. We're improving our provisioning, making it faster to get new stuff online. We migrated our network architecture from a single switch into a spine and leaf close network topology. It's really fun. It's like uh network of networks essentially. Uh, this is what everybody does as well. Uh a lot of people do this at least. Uh Kubernetes for compute density. The net result is higher engineering velocity, robust, high availability systems, and we can actually reclaim a lot of cloud spend and bring that back to our pops.
We're also going to improve the PDS hosting in some way. We're still talking about this, but uh when you have 110 servers that you rent, those servers are bare metal servers. They're ours, they're not virtual. I literally have like uh like uh IPMI login on all those things. They all fail independently, and OVH is not sending their best. And so uh as you add more servers, your mean time between failure increases, meaning your on-call burden goes up a lot. You're at the mercy of your hosting providers. When a server goes down, we're waiting for like six hours to like get notice from OVH, and it's like in the meantime, it's like okay, we can restore to a different server, or we're just gonna eat it.
And so that sucks. Uh shared storage also is a big thing that we're talking about. SQLite on the server is rough. PDS is like the best case of this. Um, but I am kind of a SQLite hater, so I'm just gonna leave it at that and we will chat. Yeah, boo you. Uh so I'm thinking about virtual PDS, the shared storage, whatever that looks like. I'm just kinda hand waving. Um and then uh more interesting PDS implementations are coming online. I'd love to talk about if you have weird PDS ideas. Uh advice or lessons learned. Uh I'm gonna start with the do nots.
Uh you must have two of everything. Single points of failures will die and you will be sad and your users will be sad. Reputation is hard earned and quickly lost. Skip the SQLite layer in your app view. Uh go with go with my personal favorite, MySQL or Postgres, if you don't like uh good things. And then uh only move past it when you're sure you need it. You know, start simple, basically. Uh don't accept local maximums. You can do hard stuff, you can do great things. That's a do not. Now do's uh first is think really hard about your data access patterns.
So like we've really optimized the absolute daylights out of the Bluesky, the Bluesky data plane. Uh you want to have this notion of mechanical sympathy, like be in tune with your hardware and like try and optimize the the shit out of it because you'll pay for it otherwise in dollars and also your sanity. Uh bloom filters are your friend, memcache is your friend, Redis is not your friend. You should have elastic compute and storage, even on-prem. You should use cattle, not pets. That kind of comes back to two of everything, right? You should have very, very thorough observability, be able to answer any question dead about your systems and do it before you have an outage.
So here's some wrecks on that. And then finally, uh one more thing of d's real quick. Build a team that's very strong operationally. You can't do it alone. And then come talk to us. Let's organize. Like Brian posted a while ago about like what does NanoG of app proto look like? Let's let's talk about it. I wanna I wanna like, you know, we're doing I posted a minute ago, like L7 BGP. Like let's go talk to each other and like figure out like the right way to do this. And there's no silver bullet. It's really hard work.
Um that's it.