Hello everybody, welcome. I think this is on. Yeah, we're good with the mic. Okay. Welcome to Coop, open source trust and safety infrastructure for all. Real quick, if you happen to have, I guess I don't know how you're gonna scan this on a laptop. Yeah, not the easiest setup. I've also been posting about this all day. But to get ready for this talk, uh we've actually created a temporarily hosted coupe instance. Important notice that you hear us talk a lot. Roost does not host our tools. We do not want your data. You cannot send your data to us.
But for today, we have some sample data from posts from all the folks in this room ingested via TAP. So if you do want to follow along, we have this interactive coupe demo. You can go in, you can create policies, you can review tasks. It's not hooked up to anything. They will have no actions and no repercussions. But you can see how the interface works. So yeah, also want to give a quick content warning. Uh we're gonna be talking abstractly about some of the sorts of harms that we encounter in trust and safety. We're not gonna like go into detail or show examples, but we will be talking about things like child abuse, violent content, self-harm, unwanted content.
Um also we will be doing, I guess we won't be doing a live demo. You gave the content warning for that live demo, but that live demo is ingesting live data from atproto. So it was. But unless some of you have been freaks, um the content should be fine because it's only looking at content posted by accounts that are actively posting on the Atmosphere conf hashtag. All right. And with that, who are we? So a lot of you probably know us. We're Roost Roost stands for robust open online safety tools. And what that actually means in practice is we're a nonprofit uh in service of the public good.
We believe that safety tools are critical digital infrastructure. We develop, distribute, maintain open source software for trust and safety, and we make uh existing safety technology integrated and interoperable. Important thing. We do not. Uh or we are not training a classifier in CSAM. We're not providing a hosted uh trust and safety platform. We're not hosting this for you, it's open and open source platform. You can host yourself. Uh and we're not doing your moderation for you. We're not a moderation service, just to get that out of the way. Uh and who are we specifically? I'm Cassidy, I'm the community manager at Roost.
My name is Juliet. I'm the head of product at Roost. Quick background, I've been in trust and safety for 10 years. Cassidy comes from a long esteemed time and career in the open source world. So together we're kind of Voltroning Frankensteining a team that both deeply understands the values and practices of open source software and trust and safety. You may recognize some of my work from the user reporting flow and internal systems at Snapchat and Grindr and Google spam and abuse. I'm sorry if you got a spam email. That is my former team. So Juliet is gonna give us a little quick trust and safety crash course.
Uh in case I guess yeah, do we want to do uh so by a show of hands, raise your hand if you are building something on atproto. Great. Now, raise your hand if you have a trust and safety plan for what you're building on atproto. Okay. Okay. Not the worst. So um let's start with basics. Um another show of hands. Who has a rough idea of what trust and safety is? Nice. Okay, that's awesome. All right, well, we're probably gonna be a little bit repetitive. Cassidy, will you kick us off, please? Sure. So a trust and safety crash course, I always say it's all the bad stuff that can happen on the internet.
But with a caveat that online and offline are no longer completely separate worlds. The harms you experience online translate into offline harms with things like ride sharing, with things like rentals, anything that happens online can also mean something that happens in the physical tangible world. But trust and safety really refers to teams, technologies, and just the general field of studying online harms, responding to them as well. Uh fun fact, this term actually was originated by eBay a number of years ago. If you look at the Wikipedia page on trust and safety, I wrote it. Uh so it has a lice nice little history about how this field really came to be and where it's going today.
It can include things like I said earlier, user abuse reporting, any kind of flagging, any kind of uh customer support flow saying something happened to me on this platform or in this experience that I do not like. It can also include things around online child safety and uh abuse and exploitation, things around CSAM. Uh a lot of people on the internet used to say CP. We as an industry are moving away from that because pornography involves consent sensing adults, children cannot consent. So we refer to it as CSAM, child sexual abuse materials. It also includes things like radicalization.
Um this can be things around glorifying mass casualty events, things school shootings, things mass shootings, things around kind of like really harmful narratives that can lead to potential mass casualty events. Things around harassment, things around non-consensual into imagery. Like I mentioned, anything bad that can happen on the internet. And a lot of the former speakers throughout the Atmosphere conference have said it also includes things around information influence campaigns. And some of the labelers that Julia talked about earlier also cover topics around things like zoophilia amongst certain communities that are very prominent and active on Atmosphere. Zophilia is still not allowed.
So with that fun slide, we're gonna move into uh more formally what trust and safety really means. And I do like the call out here of trust and safety is not just about punishing or negative effects. It is also about encouraging pro-social behavior. So there's there's a lot of different aspects. And I want to give a shout out to both Dr. Kay and Julie who talked about kind of like all kind of trust and safety work, moderation work, labeling work is inherently political. The values will change based on the demographics of those teams. Ideally, you have a team with very mixed demographics and backgrounds.
What we've seen in a lot of companies, a lot of teams, is those teams becoming very, very white and very, very former cops. That's going to influence how trust and safety is done on those platforms. But it should not be about policing. It should not be about what comes down, it should also be about what is going up. What are you putting in front of people to encourage that pro-social um behavior? And now into the academic stuff. While we were first incubating Roost, uh we did so doing research at Columbia University's Institute of Global Politics. We found that in academic studies, there was pretty much no mention of the tools that platforms actually use to do this kind of trust and safety work.
A lot of stuff about the sociology, the policy, maybe the operations, and of course the trauma and wellness that it takes on the working class people that do this work. But nothing on the technology. So we wrote a paper. We talked to over 50 different trust and safety anti-abuse and risk teams across startups, nonprofit organizations, major tech companies, and these people were very open. And what we found is that everyone's tools look the same. Doesn't matter if you're a marketplace, doesn't matter if you're a cruising app, it doesn't matter if you're a traditional social media app, all of your tools are the same.
So we came up with a framework called dire because the situation is dire. D, detection. This could be a user report, it could be a classifier, it could be hash matching. How do you know that something's up? Once you find something is up, how do you investigate it? How do you know is this the only thing that is up? Are there other things that are going on with this particular signal? So let's say maybe uh maybe there's a signal sharing group. Maybe it's literally on signal, and someone says this IP address is associated with a Russian propaganda campaign.
You want to know: is this happening on my project? Is this happening on my platform? That's where investigation takes place. It's supposed to be more open-ended. You need to understand what is the scale and scope of some of those things that are happening. Once you find all that stuff, you have to review it. Um to Acorn, the moderation tool uh built and designed by BlackSky, Ozone by the Bluesky team. Uh, we're gonna talk about Coop Our Review Tool. It's designed for scaled flexible review. How do you have the right context to take into consideration whether something's actually violating or not?
Finally, enforcement. What are the actions available? Maybe an action is taking content down, maybe it's sending them a warning, maybe it's sending them a kind of like what is it called? A de-radicalizing kind of like resource of here's some stuff to read because you're searching something that might mean something else. And this also includes reporting to different authorities. Authorities here does not always mean police or law enforcement. It can mean organizations like NECMEC, the National Center for Missing and Exploited Children. In the US, there are a subset of laws that say if you become aware of things like CSAM, you are legally required to report it to organizations like NECMEC.
And if that tiny QR code is too hard to scan, there's a big QR code for the paper. If you want to read our paper, it is right here. You can also just look for the dire framework and you should find it. Um kind of talked about this a bit. So yeah, we we we found that platforms are constantly reinventing these tools, and a lot of times they're you know it's internal tools at different organizations. Um that's part of the part of the thing we're trying to solve. I have personal. I have personally built or worked on review tools at three different places.
They all look and act the same, and they all have the same features. It's a waste of engineering time, it's a waste of resources, um, and we all have the same feature requests and bugs. This is helpful as well. Uh, these are some of the terms that we've you've probably been hearing us say, and you may hear us continue to say. Um, yeah, I kind of covered this one already as well. Yeah, and oh yes. So a phrase that we often say is roofs is tools, not rules, or better, tools for your rules, because we're not in charge of creating policy.
We get this, we get a lot of people asking us like can you write our policies for us? We say, no, we cannot. We are not only we are a lot of us have works and policy roles, but every single community is different. We're probably not members of your specific community. So we don't know what are the norms and the what's acceptable behavior within your own community. And some of this can really change based on jurisdiction. The example we have here is that in the US, um there's no mandatory requirement to go voluntarily look for CSAM.
But if you do find it, you do have to report it to Necmec and then preserve data for up to a year. Versus if you're in the UK, platforms just have to remove and report it immediately, and it's enforced by regulator that has teeth in that they will find you. So the big difference there is that data retention. Uh I've talked to you know regulators or organizations in the UK who are like, hey, like isn't that illegal if you're holding on to that data for more than a year? I'm like, actually, we legally have to. We abstractly, not we as Roost.
Um so once again, you know, I think what we're trying to do with Roost is build a community for people to share policy ideas with each other and then use those policies in our tools. And here's some resources as well. Uh Julia, I know you collected these, so I'll let you continue. Yeah, um, so somewhere on my symbol, I have made a collection of all the various trust and safety resources I often yap about. They include many of these. So these are my go-to resources if you're looking to learn more about the trust and safety field, just generally understand it.
But then also if you want to get really into the details of the operations of the tooling of the different types of technologies that are often used. Trust and safety professional association is what it is. It's professor professional association for those who do trust and safety work full time. Um that includes community mods. So they have put together this amazing curriculum written by practitioners. Um I contributed to the tooling part. There's a lot of stuff in there about how to, for example, um if like a product design run book, where if you have something like live streaming, where are the things that you need to put in friction, or what are the different types of safety tools you need to have in place before you launch something.
There's guides like that for maturity models of teams of different sizes as well as different products that you're building. They also have a library of different types of resources, academic papers, all dealing with trust and safety. Um then finally, a lot of professors are now teaching trust and safety in universities. They have open sourced their materials through this uh teaching consortium. So if you want to teach a class on trust and safety or virtually go through a class taught by experts in trust and safety, you can also do that by visiting this GitHub page. We'll share all the links in this presentation on our Bluesky immediately following this as well, because I know there's a lot of links.
All right, now we're gonna talk. That was trust and safety crash course. We're gonna talk about roosts specifically and our nonprofit and open source approach. Um the status quo that we found is really that trust and safety, it like the larger industry is kind of broken. Uh as we've said, you know, smaller platforms uh smaller platforms and newer protocols, a lot of indie small apps being built on app proto, especially, you might not know that you need these tools or they might be unaffordable or inaccessible, like the traditional industry tools that are standard. Uh and even if you can get access to them, a lot of times those application processes are lengthy and and complicated and ask a lot of questions that uh or uh an app of your scale doesn't really have an answer to.
Uh and platforms are reinventing these basic functionalities then instead of they can either use this third-party vendor or they can build it themselves. And that's a waste of efforts. It also takes away, it means that your app uh can't or your app and your team can't focus on the harms that are specific to your community if you're focusing all of your time on building the same thing over and over again. We're also seeing that generative AI and LLMs are accelerating the production of and the spread of harms. So these are just there's more and more things that are happening that that uh you know need to be addressed.
And we've also found that openness is kind of taboo in the larger trust and safety community. Um it's been really cool seeing so many trust and safety related talks here because that has been kind of pushing back against that idea, but still within the larger industry. People think that if you talk about it, then it means that bad actors will figure out ways to get around your stuff. You know, it's very security through obscurity, which, as we know from the security world, is not really a valid approach. And so what we believe is that the missing piece is actual tools, things that uh developers can take and use and build, build on and plug into their own platforms.
And importantly, more importantly, we believe that these tools must be open source so that they're accessible, uh, they're modifiable, auditable, and part of the uh safe trust and safety public commons that we're building. And as a result, Roost is very, very open source. Uh everything from not just like technically on a licensing level, but also the development model and community model that we're building is open source. So here's a number of uh open source projects that we and communities that we sort of develop. Um hasher matcher actioner is actually from Meta, uh, but we run the community and office hours.
So every couple weeks we get people together who are using HMA for hash matching and we talk about things, we we you know share information and knowledge and help develop it and make it better. Uh the Roost model community is uh this community that we've built where people who are using uh open weight models can come together and learn how to use them better in a trust and safety context. Um they can learn about new models or or different policies and how to how to plug in policies to open weight models, which is really powerful. Uh Roost itself, our roadmap is up on GitHub.
Our uh community documentation is all up on GitHub. It's all open source, it's in Markdown. You can go click a button and edit it and uh open a pull request against any of it as well. We really want to embody this open source idea and our governance documents as well. And then there's two actual code projects. We're talking about tools. Uh, there's an automated rules engine, which originated from Discord, and we helped them open source it and now is actually being used by Bluesky and Matrix and a number of other platforms. And Coop, which we're gonna talk a little bit more about today, which is uh the review console that we acquired and open sourced and have been building upon and simplifying to make it easier to self-host as well.
All of that stuff can be found on our GitHub, github.com slash roost org. And now we're gonna talk about Coop. Some of you may have known all this already, and just saying, finally, we're actually gonna see the project. So, Coop, this is the review tool. What we did as Roost, and again, we're a nonprofit. We wanted to bootstrap some of our software projects. So we acquired the IP from a startup that was selling trust and safety tools as a service, and we bought it. And then we worked over the past six to eight months to refactor that code to make it open sourceable.
We're not done. There's still a lot of really messy legacy tech that that's in this code base, but that's the fun part of working in the open. We wanted the V0 out there to be like, hey, like what's we know we need to prioritize certain migrations, database cleanups, simplifications overall. What else is needed from the community? So again, we want your opinions. So Coop, it is built for manual review. It also has like an Autobot directly built in. Um this is self-hosted on your own infrastructure. It comes with reviewer wellness features such as blurring, grayscale, and auto muting of videos.
I'm gonna go into a more comprehensive demo as well. And like I said, there's automation rules built in. There's two types of rules within Coop. One is the proactive kind of auto moderation, like you know something uh is bad or you know how to action it, your team doesn't need to keep doing it, you can set that up as a rule. The second rule, the second type of rule is the routing rule. So as things are coming in, let's say you're ingesting a million reports to your label of service or it's your moderation service, you need to sort them.
Because from a wellness perspective, if you put all those reports and tasks into a certain queue, into a single queue, you don't know what's coming up next. And context switching between something like harassment or hate to something like NSFW, cartoons, that's a lot for your brain to do. That's gonna impact you a lot more. So assigning it to specific cues, and you can also do that based on language. If you're providing services for a global set of user base, you want to have someone who's fluent in that language who has that cultural context to then go in and review.
Some of the main features. Coop is the very first free open source end-to-end child safety tool, which means it provides hash matching for known CSAM. You still have to go get your own credentials from NECMEC. Um, but once you have that, you can just plug it in and start matching against databases of known CSAM that have already been validated and verified by experts like NecMec. We also integrated it with Google's content safety API. This is not free like open source, it's free like free beer. It's a free API. You again need to get your own credentials from Google.
This is a novel CSAM detector. This classifier can find CSAM that has not yet been hashed, which is an increasing area of harm in the child safety landscape today. And we have it built in to support NecMec reporting. We've integrated it with Necmec's Cybertip Reporting API as well as well as their hash sharing API, so that way as you find CSAM, you can report it to the authorities with the right amount of detail that makes it actually actionable. It's scalable for your org. Like I mentioned earlier, you can have multiple queues for different types of workflows, different types of categories.
It's also a multi-tenant code base because again, it was originally a SaaS startup. So if you have maybe like a so let's say like you want to have multiple completely separate instances of coupe with totally separate people working in those workflows, you can also set it up like that. And I talked a little bit about the routing rules and proactive rules. There's also a unique kind of like architecture that we built in. We plug-ified everything so that way any kind of signal model detection service that you're using, you can actually integrate it directly into Coop. Coop comes with three pre-built API integrations.
Google's content safety API, like I mentioned already, uh OpenAI's moderation API, just because realistically we know a lot of people depend on it. It is free, you have to get your API token from OpenAI, uh and it supports image and text. And third one is Zentropy. You may have heard some folks at Atmosphere talking about Zentropy. They have open weight models, but for Coop, we integrated with the API, which again is backed by that open weight model for text classification. This is essentially you can design your own text classifier. You write a policy prompt in Zentropy, it creates a model for you.
It is completely tunable, completely customizable, and it's a small model that runs really well with latency. So if you need something that's happening near real time, Zentropy might be a good option for you. So for the different types of people that are building on app proto, here's where you might use coupe. As a PDS hoster, your users, your liability. You might want to review reports against your community standards. You might want to, oh yeah. Oh wow. Uh without the helper text of the elf thing, the elf thing makes much less sense. Um so there's supposed to be a line here that says in a world where sexual alfs are illegal and not allowed on online platforms, these are your obligations.
You might want to detect known images of alf from third-party databases of alf hashes and report them to the right authorities. Um as a labeler, you might want to use Coop as a review interface instead of Ozone, or in addition to Ozone, you can use automation or human review for reports on ELFs submitted by your users. You can mix automated and human decisions. And finally, as an app creator, you're like, I just don't want any of it, I don't want to deal with any of this. You can ship safety without a full trust and safety team.
We know many of you are single-person teams, two-person teams, but you also want to do right by your users. So you could use Koop to set up some simple automation. If you want to do some manual review, you could do labeling of any kind. It doesn't even have to be for trust and safety. We were gonna do a whole alf analogy and we decided last minute not to, and then we didn't take it out of everywhere. So little peek behind the scenes there. I'm gonna switch the mic. Okay.
Hope we didn't totally just mess up the live stream. You know. All right. It's on the screen here. All right, so this is what Coop looks like. For the data, I have it uh collecting content from the DIDs of people in this room. So when you first go into Coop, you see some key metrics. You get a breakdown of all the actions. I've got it set to the last seven days. What percent were automated, what percent were manual? What were the top policy violations? How many actions were taken, how many things still need to be reviewed, and you got some fun graphs because everyone loves a visual.
I'm gonna go into the manual review first. So here you can see I've got a few cues set up. I can create a new queue at any given time. And if you're doing kind of like phase review where maybe the first round reviews or maybe certain types of cues, you don't want certain actions to be taken. Maybe they're very sensitive actions. You can actually hide them from the queue. And you we use role-based access controls. So whoever should have access to a queue has access to it. Anyone who does not have that role will not be given access.
A lot of trust and safety and moderation work involves very sensitive user data. So this is a a great way of kind of partitioning things where they need to be partitioned. So I'm gonna I'm gonna go into both the Atmosphere posts and the default queue. Um I posted this, but there's really no better way to realize what a prolific poster Faye is than by running something like this and realizing all 1,183 posts essentially come from their account. So this is a post. This is essentially completely customizable. Uh whatever information you want to show, you can. Um I've just got it populated with the post that Fayin has written as long as well as their profile photo.
They did. If I go down, I can see information about their account. This is their header image. You can see it's got that auto blur there. When I mouse over it, it unblurs. They are active. This is their description. You can go down and see what other actions have been taken on this account, maybe for other posts. And I can actually preview those separate jobs directly in the UI. Finally, we're cleaning up this UI, it looks a little messy right now. But you can see additional items to give you more context to see is this account dedicated to posting something that maybe needs a label, or is it just like out of context and it's like one post and in the middle of the replies or in the middle of conversations, there's other things going on.
So I can apply a random label. One thing to know is like as you're doing this work, I believe Acorn does this too. You can actually check the policy to see what is a goose label mean. I'm gonna say, oh, it contains profanity or excessive exclamation points. This is a demo policy. There's no exclamation points here, so I'm actually gonna just ignore this task. Oh, yep, sounds good. Um so you can skip tasks as well really quickly. You can also download all of the tasks that your team has skipped. This is a really helpful way to figure out where our policies that are unclear, where are people getting confused.
Also good for generating transparency reports for Digital Services Act compliance. Umforcement, I've got a few matching banks set up. I've got hash banks with cans of spam, we know it's spam, as well as photos from Atmosphere, mostly taken from the dinners I've been hosting. From text banks, let's say you've got a bunch of posts related to the Atmosphere, you don't want to keep typing them every time you create a rule. So I've got them in the text bank. And in the rules here, you can actually see some of them have fired. For labeling Atmosphere content, I'm saying if there's any text in any item from an app proto post that matches anything in that text bank, send a human review using the policy, just do things.
There's other rules in here too, where I created some synthetic posts matching cans of spam. And you can actually see here how many things were actioned by this rule, what percentage were, and it'll actually give you that example feed of what's been triggering by this rule. You can put your policies in here. This is a really nice way to just make sure everything is in one spot. So for example, for sexual explicit, I put five subcategories. You can keep branching this out and make a very, very comprehensive tree of nuanced policies for your team to use.
Let's go here. So for routing rules, like I said, this is really looking at kind of like the incoming stuff and routing it to the right place. For this one, I'm actually using something from our signal library. I've got it hooked up the xentropy with a sexual content labeler, and I set a threshold of higher than 0.2. Nothing has triggered, thankfully, thank you all. Thank you, right? And you looks like someone added a rule. Like I said, this is a public interactive demo. Someone has put in a rule for goose art. Um so if there's any text that matches goose art, it will send it send it to the queue Atmosphere posts.
So let's see what's in Atmosphere posts. This is a picture from a dinner where I drag 30 people to Richmond. Um, and this is a synthetic post that I've created. You can see here that kind of um you know wellness feature kicking in. Because this is a hash match, this actually tells you what hash bank it came from. I've already created an issue saying it's not super clear this task originally was created as a hash match, so we're gonna make that UI a little bit clearer. Um and just to demonstrate this, I'm gonna show you what that looks like to escalate something to NecMec.
This is where the content safety warning comes in because a lot of the labels that NCMEC uses can be a little bit graphic. The UI for submitting something to NCMEC looks completely different because it's aggregating everything Coop knows about a given user to create a more comprehensive report. Necmech's not gonna be able to do much if you just submit one report for one content, especially if you're submitting 10 reports on the same user. It's better aggregate it all together. So the neckback reporting, these fields come from Necmex Cybertip API. These are the fields that their team uses to triage incoming reports to their team.
Uh the categories come from uh the tech coalition's industry classification. It essentially describes the age range of the victim as well as the type of abuse that is happening to the victim. It tells you what kind of image matched where it came from, as well as more information. You can categorize it and then send this report. It tells you what this report will look like, and as long as you have your credentials for necmec put in there, that report will go through. Okay, I know we're at timeslash overtime. Like I said, pre-built integrations, uh, content safety API, open AI, Zentropy.
I've got two labelers in here. One of the things that we're trying to do with Roost is for everyone who's trying to integrate a model with Coop, we are pushing them to add model cards because everyone deserves to know what is going on inside of that model. How was this algorithm developed? How is this model trained? What was it trained on? What are the biases? So we've got three open issues. We work directly with those model creators as part of the Roost model community to get those model cards filled out. So it's not in there yet, but it's coming very soon.
Um I know we're at time, so please take a chance to play around with our demo. I'm gonna keep it running all day and find me or Cassie if you have any questions about Roost or Coop. Thank you. Woo That's a whirlwind.