Okay, so hi everybody, welcome to our talk on transparent feeds. I'm Udit and this is uh Cole. We're from uh Haifa Worker Co-op and uh starting lab. Uh Haifa is a tech worker co-op. We do a bunch of ad proto things. And uh Starling Lab is a lab uh that focuses on questions around information integrity based out of uh Stanford and uh USC. Um I probably don't have to convince a lot of you around that misinformation and disinformation are uh growing concerns. For example, uh some of you may have seen the image on the right on social media.
Uh this is a uh an image uh claiming to be about the war in the Middle East. It's actually false. Uh the real image is the one on the left, which is a satellite image from a few years ago. And when we have both of these images uh shown together with a context, you know that the the one on the right is false, but often we don't have all the context, and it's hard to tell uh what images are real and what images are fake. And we think this is a growing concern, not only because fakes promote false narratives, but also fakes make it easier for bad actors to deny reality.
This is uh this is called a liar's dividend, and uh we think one of the things that's gonna happen with this is increasingly we're gonna see an erosion of accountability and uh and sort of assault on uh democracy. So what can we do? And specifically what can we do in the Bluesky and Atmosphere uh context. Um and so one of the things we've been working on is uh the Nectar API, which is a similar search, uh similar image search API. So you can find similar images uh on the Atmosphere. Uh one of the things we do is we run a bunch of image fingerprinting on uh in the background, so we fingerprint all images coming through uh the fire host.
And uh on top of that to demo exactly how you can use this in a misinformation and disinformation context, we've built uh a browser extension called Pollen, which makes it easier uh for us to attach provenance claims and add context to images. Um I'm gonna hand it over to Cole to demo what this looks like in practice. All right, so here's just a quick uh video demo of pollen. Here we're on BSky.app, the regular website, and we have our Pollen browser extension loaded into the browser. And we're looking at an example news account that we created for the demo called News 123.
And in this demo, they post you know headlines, stories, they post uh photographs. Uh so we're about to see them make a post, which we'll see in one moment. Uh and the post is about the Chicago River being dyed green for St. Patrick's Day, which just happened this past uh March 17th, and they're gonna attach an image from one of their photojournalists of that happening. So far, this is just a regular Bluesky flow and nothing has been modified by our extension. There's the image. They're gonna post it and it goes into their PDS. But when they post that image, if they hover over it, they're gonna see in the corner this claim button.
And the claim button is a button being added by our browser extension where any user can make a claim on any other image on the network and they get this text box where they can enter more information, more context on the image. We call this like a claim. And so um in this case, news 123 is gonna add a claim on their own image. They're gonna say something about their own content. It could be any user's content, but in this case it's their own, and so they're gonna put just a little piece of text in there.
They're going to describe the content of the image, say, you know, this is the Chicago River. We took the photo on this day, it was taken by us, news one two three, and so they're just adding that context to the image itself. And they click create, and immediately we see, thanks to the extension, one claim, and we see this drawer here that's been created by the extension where you can see the account that created it, and you can see the um the text of their claim. If you open the claim and look at it, you see it's not a Bluesky post.
It's our own kind of claim record and it has extra data in there and has a perceptual hash, it has the text, it is a link to the original post, so we have all this important information that's attached. But when this really comes into play is when you see it in other contexts. So we're gonna switch to a different account for the demo called Mallory. Mallory gets her information from all kinds of other sources, maybe Facebook and Instagram as well as Bluesky. Maybe she wants to misinform people like maliciously. Maybe she's just misinformed herself. But in any case, you know, she's writing her own posts.
She's writing Toxic Spill turns a river neon green in London, a totally different idea. And she's attaching her own image. This is a different image that she's going to attach coming from a different site. It's got a smaller resolution, it's more compressed, which you can see a little bit on the left side there. The visual content is obviously the same, but the image bytes, the C ID, it's totally different. And so she's gonna post that to her PDS. And what we see immediately, one claim. So that's the Nectar API and the Pollen browser extension carrying through this claim information because it's matched these visually similar images.
And so we see a different idea that it's the Chicago River that's been died. This contradicts Mallory, and as a viewer, as a user, we get to decide who we trust, news123 or Mallory. We also get some extra information. We get to see the user name, and we get to see 95% match. That's our matching algorithm at work, just telling you like how close these images are visually. 95% is pretty high, and so you know that's great. We also see view claimed post, which is a button that takes you to the original post the claim is on, so you can trace the provenance of the image.
You can find the original um place that the claim was made and allow you to see like you know, if you don't trust the matching algorithm, you can see what the actual original image was and all that kind of stuff. So that's just a simple demo or extension. We think the really interesting thing here is that we've now created the ability to create records that refer to images themselves, not just records that refer to posts, like when you make a reply on the network. So if we move to the next slide. So yeah, that in Pollen we've just created text claims.
We think there's a lot to explore here with different kinds of claims. You could do a geolocation claim, for example. So users instead of writing text, they put a pin on a map and they're claiming you know this photo was taken in this location. You could do timestamping claims. Every record already has a timestamp, but if you want to prove when an image was taken, you can use external uh parties to do that. You can register the image on a blockchain like the Cardano blockchain, put that registration in the record and that proves the image existed before a certain point in time.
Or you can use a third party with a trusted time stamping protocol that signs the hash of the image, you insert that signature. Anyway, there's lots of different things you can do. We think this is like an open standard that we've created that people can explore. You could do like voice notes, you could do all kinds of different cool claims with Pollen. So just to recap, we've created Nectar, which is an API to find similar images across the Atmosphere with perceptual hashing, which I'm happy to talk about afterward. Um and then we created Pollen as an example demo.
But we think there's lots of cool other use cases that you can do with this API. You could track image reposts across the network. You could use it for trust and safety to do moderation of similar images. You can do social research like uh the previous talk where you track like um, okay, this image is been has been like posted this many times over this time frame. You could do community notes. We see pollen as like step towards the idea of community notes on Bluesky and on the Atmosphere at large. And you can also do rights management to track copyrighted content or just content you didn't want to be post posted.
Excuse me. So that's Nectar. You can check us out at Nectar.hyfa.co op. If you're interested in adding image annotations to your application or image search to your application, if you want to collaborate with us on this project, fund the project, partner with us, either reach out in person to Udit or myself, or check us out at Nectar.hyfa.coop. Thank you very much. If we have time for questions. Thank you. I see the screenplay folks are right here. I know ELI recently published a blog about using Suma and Moxle some standards to try to ensure hashing of videos of all authenticity.
Along relatively similar lines is this. I'm curious if you look into those standards too. Do you think the UX for images and video can be similar in this way? And you might want to develop your extenders for the question. Yeah, yeah. So just to repeat the question, it's about whether Eli's um standards around video hashing are relevant to our work. So yeah, we we've actually worked with Eli uh and we we discussed the standard and everything. The the actual technical details of how this works, uh, we're using perceptual hashing, so it's slightly different than the idea of cryptographic hashes that match.
We also did create a standard for perceptual hashing, which is uh with Dazzle, uh Robin who's working at Dazzle there. Yeah. So uh but it's still relevant, right? Like maybe less so uh with exact hash matching, but like there are hashes that apply to video that could be used to match similar video content. We started with images because that's like the easiest lift, but you could expand it to images, I mean video, audio, and and so on, text. Uh so it's definitely applicable. Okay, thank you very much.