Thank you. So I'm talking about two years of Skywatch Blue. And I'm going to try and talk a little bit. The mic isn't picking up your voice, just a heads up. Oh, it fell off. Thank you. All right. Let's just try clipping it to my shirt. All right, how's that? Thank you. All right. So I'm talking about two years of running Skywatch Blue. And I'm going to try and impart a few lessons learned for community labelers. And some of these are hard won lessons for really anyone who's looking to maybe start a labeler but doesn't have a ton of experience both with the tech, but also I think with sort of uh traditional trust and safety.
So just by way of introduction, as Nick uh introduced me, I'm Julia. You also may know me as Scoyatolo, and I am not sure why that slide didn't advance. There we go. And I have been for the last two years running uh Skywatch Blue. Now I'm gonna preface this by saying that this isn't exactly the talk that I set out to dis you know share uh you know when I proposed this talk. Uh this weekend, you know, talking to folks, uh I've been constantly tweaking the presentation, and if I had time uh following Dr. K, I probably would have made some changes because I learned some things during that talk that apply to some of the things that I'm gonna say.
So what are labels? Um labels are metadata. They're this is the sort of actual definition provided by the um by the AT protocoks. Um it's kind of it's a long explanation. Um but the two things uh sort of to keep in mind about them is that uh they're a form of metadata about any account uh or content in the ET proto ecosystem. They sort of exist as freestanding objects and they can uh be used for moderation purposes, but they can also be used for other purposes, and there's quite a few labelers out there that are sort of novelty or fun labelers.
All right, so I have an issue here where uh the slides don't exactly advance with sort of uh part that has the notes. So if uh it takes me a moment to move between slides, I apologize. So what is Skywatch? Um it is a general purpose labeling service. Um it's one of the largest, well, it might be one of the largest community labelers on Bluesky. There's no actual way to measure that um other than by who has sort of clicked the heart button on the labeler, um, which doesn't subscribe you to the labeler, by the way.
Um it's possibly a cautionary tale, and it is definitely a personal project that got out of hand. Um the picture of uh Fred Thompson in A Hunt for Red October, where he's doing the this business will get out of hand, it will get out of hand, and we'll be lucky to live through it. Um I would also like to note, and I will come back to this in a couple of slides, Skywatch is an inherently political uh project. Not exactly in the way you think, but in the sense that all labelers are political. So what is Skywatch not?
Well, Skywatch is not a community. And this is worth noting. Okay, I don't know what that's about. Um this is worth noting. Um it is a community service, but one of the sort of uh key things to remember about Skywatch is that it started with a few dozen users in mind. Um and it grew at a slow rate until November of 2024, paralleling site's growth, at which point it exploded by an order of magnitude. Um I guess to that end, it's also hard, you know, it's hard to say that it serves a specific community. Um and one thing I haven't cracked is how to is what to do about that.
Um if you start with an original community of you know a few dozen and now you are up at potentially you know 8,000 or more users who come from maybe backgrounds that you didn't originally plan for, um, it's hard to say how to serve them best. Um I also believe that Skywatch isn't a moderation service. We're not doing traditional trust and safety. We're also not, we don't have takedown capabilities. Labelers are metadata. You can have block lists associated with certain labels, which we do. But block lists are soft moderation in the sense that they don't necessarily affect, they don't affect takedowns.
There was a recent paper actually published which talked about the fact that they don't really break up the social graph either. And that's probably a good thing, even though the block on Bluesky is understood to be a nuclear block. I will say that most labelers are not hard moderation. Some exceptions would be uh you know BlackSky, uh North Sky, and of course Bluesky moderation. So then work. By way of example, um, I just wanted to show some of the labels uh that we sort of append to content. Uh I've broken them out by account versus post.
Um and when I say that we're appending metadata, that's exactly what we're doing. Um we're analyzing the content in one way or another, some of it via automation, some of it via manual review, and then we are applying a label based on the rules and the definitions of the label that we provide. So I do want to, before I go on further, go back to the notion that SkyWatch is an inherently political project. And to do that, I want to talk a little bit about why I started it. Um in 2018, I co-authored a paper with Kate Starboard uh and her team at the University of Washington that looked at the media ecosystem around the Syrian civil defense, uh, also known as the White Helmets, uh, in Syria during that country's civil war.
And we started with Twitter. Um, some of her team went on to analyze other platforms. And what we found was that there was an active attempt to reshape the information space through content sharing and remixing across multiple sites, domains, and political leanings to create the appearance of independent confirmation. The goal was to contest the civilian status of the white helmets and legitimize their targeting under international law. Skywatch came about because Bluesky offered a means to shape my personal information space and the information space of anybody who chose to use the labeler, which at the time was a small handful uh, well, a few dozen people.
And I think it's important to remember that all labelers and content moderation do this to some extent, and and you know, whether the the admins of it are conscious of it or not, but we should all we should be honest about that. Um so I've always tried to be open about my motivations, just that Skywatch is opinionated. Um it embodies a sets of values and ideas about what users want to see on their timelines, and so this is what I mean when I say the project is inherently political. However, there are a couple other reasons why I started it.
Jerry nerd sniped me. So the first label was really the Bluesky Elder label, um, which follows basically the criteria laid out here. Um then uh next is and content warning, but my partner uh who's in the room created this monstrosity and it became part of early Bluesky culture, which is why we have a sensual alph label. Yes, there are community standards, and I am trying to live up to them here. Alright, so you want to start a label, uh, a labeler and a non-novelty labeler at that. I think the first thing I'll say is don't.
If you do, you're probably someone who enjoys getting yelled at. Um you're now a mod, whether you like it or not, even though I said we're not a moderation service. Um every label is going to be perceived as a personal attack. I mentioned you're in yellow de. Which, to be fair to users, we don't have, if you're not hosting a PDS and having that labeler associated with the PDS, you have no way to do the two-way communication for you can get appeals in, but you can't explain to people why they were labeled in a private manner.
There's no way to send them an email. And that's a shortfall of the tooling. Um folks don't read the label definitions except for the rules layers. Uh, and uh they're always going to be you know trying to to sort of you know work you to get the rules uh sort of either either uh you know uh talk you into removing a label to uh from them or to apply a label to someone else because they think it fits your definition. Um and I think a lot of folks, this is getting a little bit better, but I think a lot of folks don't understand what community labels are and can't necessarily tell the difference between you and Bluesky.
So one thing you're going to have to learn is that I'm not sure. I'm sorry. Um your signals are noisy. And there we go. Um user-generated signals are noisy. Um you are to a subset of users going to be a gun and they are going to attempt to weaponize you. Um but as I said before, lots of folks don't necessarily read the labels, and as such, um they're going to submit a lot of reports that have nothing to do with the things you label per se. Case case in point, um, we are constantly getting reports of things that people suspect are AI.
Now they are not necessarily misrepresenting what they are. I discovered this morning people had started reporting ADI. Um it's just a fact of life. Umser reports are noisy, automation can fix some of this, but automation is also capable of making mistakes at scale. So there's there's always going to be risks inherent in either approach. Um, in conversation with Luca and Juliet earlier this weekend, uh, one thing that did come up is that there are ways to mitigate this through tweaks to the reporting workflow. So Brian, I'm sorry, you're probably gonna get some GitHub issues from either me or Luca uh you know requesting changes when you all have the time.
Um this is difficult if you don't control the reporting workflow. Scale is very hard. Um if we go back to the fact that yeah, I know. I went fully insane and I think the website probably uh you know uh was going through some things. Although interestingly um uh uh in well, I'll get to it in a minute, but um remember, so growth can be a bit of a curse, especially for general purpose labeler. Um you know, one of the things that makes this difficult is um not having a specific community that you serve makes it a little bit harder to find and onboard volunteers.
Um we had an eternal November that well, not not eternal. It lasted about five months. Um we were getting uh so November was 70,000 reports, November 2024. Um and interestingly, I was talking to Brian, and this was actually different for them because we don't deal with things like sexual content, we didn't have to deal with the flood of reports that came in when video was rolled out. Um so it's potentially unique to each labeler and kind of what your labels are and what people perceive that you are covering. Um reports began climbing you know right after uh election week uh or election day.
They peaked in November uh 20th or so. Um and then there was a steady decline over the next uh four to five months. One thing I think that helped was that Bluesky made some changes to the reporting workflow so that you could no longer sort of uh shotgun report content to every single labeler you subscribed to. It forced you to be a little bit more nuanced. Um we've now settled out at a baseline of between three to seven thousand reports a month. Um it it varies uh and I suspect that as we get closer to midterms, we'll probably see a spike there as well as people start reporting certain content to uh Skywatch.
Um the other thing to keep in mind is that you're going to create new affordances. Um and some of them are gonna suck. Uh and you should undo your mistake. So this is actually not a screenshot of Skywash. Uh this labeler would put a badge on your account if you followed one of these individuals, with a theory of change being that by making this legible, which is what labels do, they make certain things and behaviors legible. It would cause users to shun certain accounts or and some social pressure to for folks to unfollow those people. Unfortunately, it facilitated abuse, which is why it no longer exists.
So you're gonna make mistakes. Some of them are going to be innocuous. Like slurs, for example, are hard to do, to sort of automatically flag. You'll decide one day that you want to label uh an unfortunately far too common English language slur, uh ableist slur, go to bed and you're gonna wake up with the francophone world mad at you because you forgot that it's a benign vocabulary word in French. Um so you need to approach these things with some nuance, and fortunately we have the tooling to do so though imperfect as it is, we do have the tooling to determine you know what language a given post is in.
So you're gonna want to label behavior in facts, not identity, um, and you're gonna want to do it consistently with your label definitions. Um I think it should go without saying, but don't libel people. Document your claims, uh, you know, be ethical. Uh you probably want to check your renters or homeowners insurance to make sure that if you're not a business, this is to make sure that you have uh just a sort of universal umbrella coverage in case somebody decides to hit you with a slap lawsuit. You are generally covered, at least from the conversations I've had with lawyers.
You are generally covered under section 230. If or not. Don't build up to exist. Yes. Yes. Thank Ron Wyden for fighting the goodbye. Um, you know, your mileage may vary by jurisdiction, of course. Uh and and um, you know, there's absolutely no certainty, you know, in uh climate where speech is generally highly politicized. Um labelers can be fun, and I you know don't want to just talk about the negative parts of it. And so there are affordances of labelers. And this is not really in Skywatch's remit, but I was testing whether or not I could use Osprey's database to figure out who is the top Taylor Swift listener that day using you know using the Quick House database and using teal.fm's lexicon.
And I could. And it was pretty simple. Um, you know, it's kind of funny because then you can kind of go from there and independently invent something very similar to Addy, where you take the posts of the top TLFM listeners of Taylor Swift tracks, and you just find some of their posts, and and now you have a feed that you could potentially create. Again, not really within the scope of what we do, but I do want to sort of point out that labelers can be fun. So looking a bit to the future, um, I want to talk a bit about some of the tools available to make this a little bit easier.
So if you follow me on Bluesky, you've probably seen a graph similar to this. This is a graph of the sort of the the the Gaza spam accounts that have cropped up in the last two weeks. The accounts where they are recycling the same set of usernames or the same set of um profiles and you know doing things like reposting people so it just shows up in their mentions. Um it is generally a good indicator of inauthentic behavior, and we're labeling it as uh as inauthenticity. And I know that that this specific case is controversial, but this is more by way of example.
To do this, I would never have been able to do this or detect this at scale just using Ozone and manually labeling things. So we can make it suck less by using better tools. Um Romod started as a bash script. This was two years ago. Um that lasted about a year. I rewrote it in TypeScript, and now we use Ospre. Um it's great. Um we also use uh uh it's Osprey plus Ozone. Uh and then I use Claude and a stack of skills and tools to help me rapidly query the database. So Claude is not making now, you see here, Claude has opinions, but Claude is not making judgments or content moderation decisions.
I've actually experimented in the past with trying to use LLMs to assess individual pieces of content, and there's just not enough information there, at least for the tools I was using. Now there are probably more sophisticated uh tools, there's fine-tuning you could probably do to to get your accuracy up. But for the most part, what I find it extremely useful useful for is going out, querying Ozone, querying the Click House database to put together reports that I can then dive deeper into to see is this really a signal or is this just something spurious. And in this case, I have uh to illustrate it, you know, there's basically uh a small network of accounts via you know coming via Bridgy Fed that are uh sort of reposting what you would consider you know uh Russian military uh telegram copypasta and uh sort of other uh you can call it propaganda, um, other content aimed at various different language groups.
Um it's a relatively small number, um, but it's there. And um, you know, if you uh have these tools, it's relatively easy to find. Um, you know, and to that end, I've been building some of those tools, such as sidecar services to do you know croissant analysis on PDS signups, um, you know, looking at URL over dispersion, uh doing some lighting clustering and account entropy calculations to sort of help detect that activity. I'll say that those are noisy signals, so by themselves they they shouldn't be sort of relied upon. So human review is still important and correlation is still important between sources between signals.
And I just want to end on the note that automation is not perfect, it can make mistakes. Um, but it's a lot easier than trying to find these things just using the regular expressions or just using searches on Bluesky. Um and since our users subscribe uh for uh especially for some of the the the sort of in office you know authenticity labels, um, it's important for us to kind of proactively search for these things. So what is the future of unintegrated uh labelers? And by unintegrated labelers I mean what Skywatch is. We are not affiliated with uh set of PDSs or an app view in the way that Bluesky is or that BlackSky is.
Now my theory is that the Atmosphere because the Atmosphere is growing, that not every application or PS admin or relay owner is gonna want to set up their own full stack moderation service. Um they're gonna need trusted signals to action content and accounts on their network. A great example, excuse me, is PDS admins, and I want to give a shout out to Bailey for building PDA for building label watcher. Uh this is it. If if you haven't talked to Bailey about it, this is a tool for a PDS admin or any other piece of infrastructure to watch the fire hose coming off of some lay a set of labelers, and then it can either email you to tell you when a label of interest uh is detected uh being emitted against an account on your PDS.
Or it can automate the automate the takedown. Um the nice thing about that, it is up to you as the admin to make a decision whether you trust the labeler enough to actually do the takedown. Um so as a result, we have this new group that previously wasn't considered a user that is now a user. Um and so Skywatch is one of the signals for catching spam accounts created on self-hosted social and and VMX uh social. So thank you uh Bailey for building this and also for placing your trust in Skywatch. It's all because of SkyWatch.
I think I think I think we nerd sniped ourselves with that one. Oh we did. So now I do just want to talk about this a little bit. Um because we've just come up, I just the previous slide we talked a little bit about like how Skywatch can be used by by PDS admins. And I want to note that labelers are an informal governance institution in a sense in a commons that you know, as it grows, it's gonna lack formal governance over the whole commons. Bluesky is going to take responsibility for certain parts of it, other parts of it, other folks will, and other parts of it may be somewhat ungoverned.
We should remember that labelers are embodying sort of a form of power and authority. Um, they're agents which act on the network. Um we should also note that some of these tools are incredibly powerful surveillance mechanisms, and so they need to be used ethically. Um we need moderation, but that by its nature is surveillance. And so seeing like a moderator means trying to take behavior and activity and making it legible, and that creates the temptation to fence in the commons, categorize people and behavior in a binary fashion, and then tightly define what's right and wrong, as Dr.
K sort of had uh alluded to. Um sorry, no, that was um a previous session in here that was uh Dr. Blackwell. Um we don't want to fence in the commons entirely. We want to be able to sort of set norms uh and uh set the rules to govern the parts of the commons that we care about that are within our jurisdiction without trying to restrict how everyone else can act. Um, all right, I am very close to wrapping up then, so thank you. I do want to talk about the elephant in the dark forest um because I'm an independent labeler.
Um if you've not read or watched the three body problem, the dark forest metaphor uh is the idea that um all advanced galactic civilizations are locked in a bit of real life game theory, which the annihilation of any other civilization that they encounter is the only correct move. Um this is a kind of a tortured metaphor because I was going for the elephant in the room plus the dark forest, and I don't necessarily want to cast dispersions on elephants. Um but I think we do need to talk a little bit about Aegis. If you don't know what Aegis was, it was one of the first independent labelers.
Uh the operator ended up using it for harassment, and I think it soured Bluesky, both the community and the company on the community labeler experiment for a while. And I want to remind people that I, as an individual, sometimes with help, sometimes with not, am running something that has the capability of being a surveillance platform. Labelers collapse context. And at scale, they could potentially break the social graph. And I think it's great that people put their trust in me. But by the same token, they shouldn't have to. You shouldn't have to trust that Fane will take away the keys if I suddenly go rogue.
And you know, uh, it's it's a bad model. Um in a previous life, I worked on issues of data ethics and privacy and the harms technology can cause in humanitarian response. Jay quoting Melvin Kransberg yesterday, I think was important. These tools are what we make of them. But we can't assume we're the only actors using them. And the assumption that we are acting in an ethical and just I want to think that I am. I want other people to think that I am, but I don't want you to have to rely on just that. And I think that we probably have to have some conversations as uh sort of a community about how we govern the commons and how we govern labelers.
So I would like to encourage people to come to the discourse and start discussions there and move us towards a composable moderation working group so we can set some standards and have some of these discussions. And I just want to wrap up with this quote uh from uh from Lawrence. Oops. An open protocol that does not answer the governance question will still receive an answer, yet it'll be written by whichever actors make themselves indispensable first. And then I do want to finish on the note that I do think labelers are important. I think that they're incredibly useful.
I think we have the opportunity to shape the spaces that we are sort of inhabiting for good. Um, even though this talk has had pessimistic notes, um I want to end on the note that you should check out some of these labelers. Skywatch is there, of course, but LilAps, Stash Lab, and BlackSky are also labelers that I use. I find valuable, and I hope you do too. Um I think we are probably just at time, so I don't want to run over, but if there are any questions, I'm happy to hang out for a little bit and answer them.