Alright, that's like 24 of the time. Alright, uh welcome to my talk. I'm presenting crowdsourcing research synthesis on atproto, envisioning an inclusive feature. And I'm Jay Patel from the University of Maryland. So I want to preface this by the motivation for this lightning talk. Back in summer 2025, I was doing my first systematic review. So who's familiar with the idea of a systematic review in research? Okay. Who's familiar with the idea of any kind of review research synthesis? Very good. Okay. So systematic review is a very systematic way of sort of assessing how much evidence there is out there in the broader research literature.
And that involves four key steps, which is searching for relevant preprints and papers through keyword searches and natural language searches, screening for relevant reports, so from all the papers that you search for, which ones are relevant to your questions, and appraising or evaluating the study quality. After appraising for study quality, you might drop off filter out a few more articles. And then when you do that, you'll then synthesize the evidence that you have into some sort of prose uh which is suitable for publication. So this is what I've been working on for the past year. And the question that I was really trying to answer was how suitable are large language models for assessing research reports.
So as I was doing this, I noticed a few recurring problems that uh are not exactly novel, but I really felt them in my bones. Uh one is just that it's very slow to do this. It takes about one to two days to uh read a paper very deeply and to be able to say something smart about that. Um there are many such citations for that claim. Um it's also very tedious um after you learn it, after you go to the first dozen or so papers. Uh and if you want evidence of that, you can just go to me because I've done this you know hundreds of times.
Um this is actually a very costly procedure. So if you have a trained graduate student and maybe one other person working on it, um, it takes quite a lot of money. Um and there are only a few citations for this sort of thing, but estimates range between 50 and 100,000 USD. Um actually in pharmacology or a pharma uh companies, it'll cost about 141,000 per systematic review. So that's quite a lot. Um, and sadly this is also an error-prone sort of uh thing. Uh 93% of systematic reviews have at least one error in the search process. About 59% of the sampled ones have at least one error in the extraction process, extracting key details of methods and analyses.
And then when it comes to appraisal or evaluating the study quality, well, uh there are very many problems, and I would call that foobar, or I guess you know that acronym. Um there's also a number of other problems, like um you use very often small homogenous teams, which limits the quality of your systematic review. Ideally use very large heterogeneous uh teams uh so they could get through the work more quickly, uh tackle the problem from multiple frames using different kinds of knowledge, different kinds of skills, and use different connections to make sure you have a really great product at the end of it.
Um even more problematically, many of these systematic reviews, because they take so long, are dead upon arrival. Um they're rarely updated after publication and very troublesome to maintain even when you have a fancy sort of uh living web page. So what we would ideally have is uh you know solutions to these four key problems, volume and velocity. Uh we would increase the speed at which we do reviews so that they take weeks, not years. We would improve the veracity or accuracy. Uh and I propose that we do that with uh crowdsourcing, increasing the team size and ensuring that certain team members are always engaged with error detection and error correction.
Uh we would hopefully increase the diversity of the different perspectives and contributors on our teams. Uh that would allow us to reframe things, make suggestions, make interesting connections. And then also have living systematic reviews, which helps with uh providing timely data to people who might benefit from that data. And we kind of do this thing already with open source software development, right? So how do we do this? Do we use some kind of shiny app with R, open science repos? Do we use like journal-specific tools or journalware? Uh this question has been asked on Bluesky actually quite recently by this user who asked, what is the best platform to host something like this?
Meaning a s living systematic review. Is it shiny? Is it a website? Uh well, you know, one other user had this interesting idea of uh a wiki, which is a more dynamic approach where multiple researchers can edit and attach new evidence and reviewers, uh, and then various other people can make uh approvals, rejections, and modifications. So something like Wikipedia, but with additional guardrails. Uh the nice thing is that the atproto system is kind of like a full-ish stack right now. Um we have all the sort of necessary tooling to do an MVP. So suppose you have a bunch of research posters, a bunch of preprints, a bunch of fully polished papers, and then you sort of discourse graph them with their questions, claims, and evidence, and push those data into symbol collections and more uh merging annotations, then you could probably layer that on to some kind of UI where uh users can validate the different uh extracted uh discourse graph nodes, agree, and then publish that into a leaflet blog post, um, which recently has been working on um uh creating sort of living documents, living posts.
So if we do this, you know, I think the near-term goal could be to leverage the existing apps, create the MVP. Hopefully that can be done within a few weeks to months. In the medium term, I think it'd be cool to query data across multiple crowdsourced reviews and scaffold also what's called primary research, uh, which would be not reviews but primary experiments, surveys, interviews. And in the very long term, it'd be very nice if we could build uh interesting visual dashboards and data stories that uh would appeal more to the public, uh, more for public engagement, uh, rather than just creating you know typical research reports for scientists.
Uh so there are a fair number of things that we can do, uh, and I'd like you to kind of get your help uh with this. So if you're a builder, can you help me build the missing pieces? If you're a scholar or someone else, you know, can you help me test drive the MVP, spread the word? And I don't know who else is out here except a builder or a scholar or uh advocate, but anyone else, you know, you can also help uh test drive the MVP, spread the word, get involved. Um would really kind of love your advice here.
I'd like to thank my research mentor, Joel Chan, for uh conceptual guidance here, and then Ronan for also motivating this talk and bringing me out here. Uh and finally, I'm available on Orchid and Bluesky. Thank you. Thank you, Jay. Sure. Uh questions, anyone. Uh maybe I'll ask a question about the Yeah, come just comparison. Like what's your sort of mental model for where atproto fits in versus like AI? How do you see the two combining? Uh I'm agreeable to both approaches. Um I haven't thought about AI as much because uh I think simply increasing the team size would be such a great benefit to uh the problem at hand.
I know that for systematic reviews, there's an incredible amount of work uh being done right now for screening papers, extracting bits of uh questions, claims, and evidence, and also doing the synthesis. I agree more for using it for search uh screening and uh extracting questions, claims, and evidence. A little bit less so for synthesis because that's way more complicated and way more fun. As an amateur, I've tried to sort of do that myself and it's not been as successful. Um but certainly I think you know, search. In fact, I couldn't have done part of my systematic review without using some AI-assisted uh elements into my workflow.
It really would not have been possible. So I strongly advocate for using the right uh AI programs to help with search and screening. Um, although my systematic review is so small that uh I can do the screening myself manually. Um and then when it comes to appraisal, uh I have a certain way of doing it that I like, and so I feel comfortable just doing that myself. Um, but I'm open to anyone who might be able to sort of automate custom built checklists and things uh using some sort of uh AI agent. So that is yeah, it's quite doable.
Cool. Anyone else. You mentioned uh using high quality LLMs. Did you benchmark maybe like mid medium level or low quality ones and change it? For the systematic review that I was conducting, I did not do the benchmarking myself. I was merely reviewing the studies that others had done, the benchmark experiments that others have done. And so what I noted from that, which is actually helpful for this uh sort of meta project is that you know there's a smattering of results, the more you focus these uh foundation models uh the better they do. Um fine-tuned data, better prompts, better context engineering, those things all sorts of help.
Um but it I I think none of these models achieve uh you know expert human performance on every trial, so uh it would really be beneficial to have humans and AI agents work together across the systematic review process. Certainly I see them being very useful for appraisal, right? Let's say you have a long checklist uh that's useful for evaluating the methods or analyses. You should you should consider certain LLMs, I think. That's perfectly fine. Yeah. Any more questions? Maybe a quick one while uh Sophie sets up or we'll just move to the next talk. Okay. Well let's thank Jay again.