Hi everyone. Welcome to a talk about MataDisco, which is very, very disco. The first thing you need to know is that you might think of me as Robin Burgeon. I don't know who has been spreading that rumor. My name is actually Volcker, and I am giving that talk because I am the creator of MataDisco. Are we clear on this? Yes? Okay, thank you. So, you know, one thing that's pretty cool about have you heard of the AT Protocol? Some of you call it atproto for some reason, is that it does like social stuff, but it's not just social media.
And in fact, science is itself a social process and an entirely democratic one. And so we should also use social internet for science, and we could disco with data. What is the problem that this is looking at? Well, it turns out that a lot of what happens around science, apart from the science itself, sucks. Data in the science universe is pretty bad. There's a ton of really cool data. Lots of people are sharing it in the open in all kinds of way, but like the it's all siloed, it's really hard to discover, a lot of the servers are terribly slow, the the APIs don't work, the portals are designed by I think grad students, maybe not satisfied.
So you know, some some of those things are pretty pretty bad. And and one of the big classics is of course like all these scientists putting all their data on GitHub because we all know that nothing bad happens on proprietary social media platforms, right? Um so anyway, we're trying to like, you know, the um Tom Nicholas from Frost wrote this like really cool uh blog post about it. Um it's important to think about this because you never know when you're going to need it. So one of the things I've worked on before, I was a small part of the New York Times team that collected data for the you've probably sadly seen this, the COVID tracker.
Um and what we had to do for that to work was scrape data from like all over. Um and that actually that the team got a Pulitzer Prize for web scraping, which is quite interesting in and of itself. But like the the the level at which this data was bad is terrible. So like Italy, for instance, did a good job. They put all their numbers on GitHub, it was really easy, it was JSON, etc. Um in the US, most of the time, data was per county. Um and those counties all had different systems, so you had to have different scrapers per county at the US level.
Um one of my favorite ones was France, which I worked on a scraper for, where like every day someone would get all the numbers from all all of France, paste them on a JPEG with like a nice textured background, and put the JPEG on the web, such that scraping that involved a lot of processing to clean it up enough to OCR it to get the data out. And so this is the kind of situation where you do need stuff. Um and like more generally, even without COVID, you know, like if you imagine the impact of making research better, like every time we make it slightly better, scientists can science more.
And so what does like good look like? Well, so in that blog post I I referenced earlier, like you can think of like a number of properties that would be pretty cool to have in in you know sort of like the scientific data space in terms of discovery of of data repositories, data sets, etc. Um, you want it unified, everyone can find it, of course it's free, it works, it works on any scale. I'm not gonna read through all of it, like but like all of these properties are pretty interesting. And Volcker has made a tool with the IPv Foundation called MetaDisco.
And what MetaDisco does is extremely simple. It tracks data that gets published to all kinds of different um data sets and data sources. Currently it's limited to just a few because it's still like ramping up. But the idea is to put all the metadata around all everything that happens, everything that's published directly on atproto, such that then you can write apps that just listen to uh whatever jet stream, whatever's whatever's convenient in terms of fire hoses for you, and then react to that and do things with that data, pull it down, etc., knowing of course that um all of this is nicely verified.
And so if you if you think of the list of properties that I produced earlier, well, it hits quite a number of them, right? You have something that's inherently social, you can subscribe to it, you have a right to exit because it's all on a PDS that's part of the protocol. And you know, sustainably fundable, well, it's not very expensive, um, but like we could we could figure it out. Um and and there's a few properties that it doesn't completely hit, but in general, like simply by pushing all this metadata about data sets to the to the protocol, it's getting a lot of these things working already.
And so um I made you know a very simple uh demo of this. You can go on my GitHub, VMX.io slash Matadisco viewer. It's all on the Matadisco.org thing. Um, this is just like tracking um satellite images as they get published as they get pushed out, and they come in directly on the fire hose and you can display them directly. Of course, this is not super interesting in and of itself, but um we can we can build stuff on it. And you know, we're currently in the process of integrating further sources. This is like stuff from uh triple IF.
And so you know, data side data sets, when we talk about scientific data sets, people immediately think like satellite images, genomics, whatever. But like, you know, they we also have like museum pieces, you know, cultural things, like um IIIF can do like sync between video and like um uh notation stuff. And so yeah, all of this um brought together, go to Matadisco.org. It's brought to you by the lovely IPFS Foundation. And uh Ted has a question. Um hi Volker, thanks for this. This is really cool. Um I was wondering if you'd seen the sort of lensing work that Nick Darkinus and uh Blaine Cook and other folks have done.
Um because if you get all this data up on protocol, like being able to quickly translate into other formats would be wild. So I haven't seen that specific thing, but I I I knew about the old Cabria stuff, and Blaine has been what's the nice word for harassing, encouraging me um to to to have a conversation with him, which is always a pleasure, of course, uh specifically about that. So I yeah, I really want to look at it because uh the better we we pump the source metadata in, and of course those differ. Um so yeah, being able to lens them between between between formats would be amazing.
Yep, um uh Nick has it currently deployed actually. So let's let's let's just we can just do things. We can just science. And that's it. Any other questions? Yeah? I just had a comment. Wait, uh in in the in the mic. Yeah, I'm I'm a climate scientist, and distributing data is a huge deal for us. And uh we can talk more about it, but uh the customers who want uh climate data and it'll be really great to do that. Yeah, um bring it our way. One of the things that's great, so like as you know, there's ongoing epistemic side in the US, and like a lot of people have been trying to save data sets as fast as possible by by putting them on on BitTorrent.
What's difficult with that is you can't know if they're publishing the real original thing or if it's been tampered with because it's very difficult. I mean, there's currently no source to know um what it was like with this, this because atproto is built on Dazzle brought to you by the IPFS Foundation, um everything can have a CID, so everything can be verified, so it's it becomes possible to load data you know that you know and have like a public um description of that that gives you very verifiability built in. Um so I think this is this is a win for the next wave of of epistemicidal maniacs.
Have you thought about a T log over it? And we could probably do a T log of it, all these things compose, right? Hey, uh how much data have you uh uploaded to various PDSs? Enough to knock over Eurosky PDS. We're only pushing the metadata out, right? We're not pushing because like some of these data sets are insanely massive, we would like knock over everything. Yeah. Um but like we've pushed, I mean, I don't know, like hundred at least millions of records, I think. Um if you if you look on UFO, uh MetaDisco is regularly in the top ten lexicons, even though no one knows what it is at this point.
So yeah. Any other takers. Just go use it. And yeah, if you want to onboard stuff, talk to us. Okay, thank you. Thank you very much.