‘Use Your Brain!’ And Other Pointers From a Computer Scientist on Using AI in Research

Sound clip: You’re listening to a Times Higher Education podcast.

Miranda Prynne: Hello and welcome to Campus Talks, the podcast in which we seek advice to improve the work of those in higher education. I’m your host, Miranda Prynne, Campus editor. And this week we are exploring something that everyone is talking about, but few really understand. Drumroll. We will be discussing artificial intelligence, or AI. It’s easy right now to feel overwhelmed by the sheer volume of commentary and well-intentioned advice relating to the use and misuse of AI across higher education. That’s in teaching, in research, in academic publishing, in administrative functions. It really does reach into every facet of university and academic work. So today we are zooming in on the potential of AI as a research tool. Now, the computational power of AI has exploded in the last decade, making possible advances in research and data analysis that would previously have been unthinkable. From protein structures to breast cancer diagnoses to climate modelling, AI is driving major leaps forward in our understanding of ourselves and the world around us. But a technology as powerful and disruptive as AI inevitably brings risks, as well as opportunities. And that applies to its use in research as much as anywhere else. To find out how AI can best be used in the pursuit of knowledge while managing its limitations, I spoke to a senior academic who has been exploring AI as a research tool for more than 20 years. Karin Verspoor is dean of the School of Computing Technologies at RMIT University in Melbourne, Australia. Her research focuses on the use of AI methods to enable biological discovery and support clinical decision making. She was previously director of health technologies and deputy head of the School of Computing and Information Systems at the University of Melbourne and scientific director of health and life sciences at the NICTA Victoria Research Laboratory. She is also the co-founder of the Australian Alliance for Artificial Intelligence in Health. So, if anyone can help us to decode some of the complexities of AI, Karin can. We started by talking about how she uses it in her own research and then moved on to a broader discussion of how researchers should approach the use of AI, in particular generative AI, to maintain research quality and integrity.

Miranda Prynne: Hi, Karin, it’s great to have you with us today.

Karin Verspoor: Hi, I’m so glad to be here.

Miranda Prynne: Now, I’m very keen first for you to talk me through your current research on the use of AI in biological discovery and some clinical decision support. Have I got that right?

Karin Verspoor: Yeah, that’s exactly right. So, for the last 20 years or so, I’ve actually been trying to use the tools of AI to help organize information for scientists on the one hand and clinicians on the other. And that means really trying to use the tools of natural language processing and machine learning to dig into that messiness of language that we clearly use for all sorts of communication, but really importantly for scientific communication on the one hand and documentation of clinical episodes and consultations on the other hand. And because it’s all in language, we have to work a little bit to transform it into something that is usable and actionable and closer to facts rather than the complex communication of English or another language.

Miranda Prynne: Fascinating. So it’s almost like translating the different interpretations via language into perhaps something more objective. Is that the aim?

Karin Verspoor: That’s exactly right. So you can think of it as summarizing the information. So, you know, if you have a really complex sentence that communicates relationships between, say, diseases and genetic mutations, you want to try to boil that down to the key entities, those diseases and the mutations, and the relationships between them, so that we can put it in a database. And the purpose of doing that is multifold, actually. So, on the one hand, we want to be able to make that information easier to find. Because if you search for something using language, you often miss the core facts because you’re not using exactly the same language that was used when it was written down. So, this is really about trying to put it into a structured form that we can manage more easily. And then, on the other hand, it’s about using it for further analysis. Once that information has been transformed into a kind of structured representation, then we can use it for other kinds of modeling and statistical analysis.

Miranda Prynne: So, would this be looking at, for instance, medical notes, or are we talking journal papers or all of the above?

Karin Verspoor: All of the above. So, I actually started with journal papers about 20 years ago because that was what we had accessible to us. And it was a big challenge for scientists to navigate the full range of scientific literature. Now, we have about 38 million publications indexed in PubMed. And 20 years ago, there weren’t obviously there weren’t that many, but there were still many, many millions of papers. And scientists just can’t keep up with all of that information. And so this is really about helping to find the information, to curate the information, and help scientists kind of navigate that onslaught of information that’s in that really complex linguistic form. Clinicians have the same problem in a sense. And really, because as patients travel through the healthcare system, we are documenting huge amounts of facts about them. We’re capturing the signs and the symptoms that they present when they go to visit their primary care physician. We’re capturing reports of describing images or histopathology results or laboratory analyses that are being done on their behalf. We’re capturing the progress notes as doctors or nurses round through the hospital. And all of that is being captured in language. And it really is just overwhelming amounts of information that become very difficult to navigate without the use of a computational tool.

Miranda Prynne: Yeah, of course. But then I can see that computational tool needs to be extremely intelligent in the way that it is able to pick out and understand the key messages sitting within all those words. We will come on to that. But first, what got you interested in AI applications in this area or indeed, you know, AI at all?

Karin Verspoor: So I started as an undergraduate computer science student. And I was lucky enough that around the time that I started, the university I attended, Rice University in Houston, Texas, started offering a new degree called cognitive science. And I knew I wanted to do computer science, so there was no question there. But this cognitive science major really grabbed my interest. And it involved studying psychology, linguistics, philosophy. And I was just hooked. I just thought it was so cool. So I decided to do a double major and study both computer science and cognitive sciences. And I had a moment in one semester. I remember really vividly, I was taking an artificial intelligence class as part of my computer science major. But at the same time, I was in a psychology of language class that was talking about essentially how the brain deals with words and ambiguity and complexity that’s communicated in linguistic form. And I had this moment where I realized, wow, they’re actually talking about the same thing, but from completely different perspectives. And I was fascinated by this. And it sort of got me down this track of thinking, well, how can we use computers to help us better make sense of language? And can we use a computer to really capture our understanding in a way that allows us to then process that language more effectively. So it was this dual thing of using the computational models to understand the language itself, but then also make the output of that process useful for people in the way I talked about earlier.

Miranda Prynne: I feel that you are the walking embodiment of why interdisciplinarity is so incredibly powerful when those synergies happen or links are made. Because very few other people probably could have had those two insights and been able to match them up.

Karin Verspoor: Yeah, look, I often talk about this as the beauty of the liberal arts degree, right? That you are perhaps studying a technical subject like I was, but at the same time being exposed to other kinds of ideas. So, when you’re exposed to different perspectives on a single concept, you start to realize you find these connections between things that you otherwise wouldn’t be exposed to. And actually, for me, that has set me up for a highly multidisciplinary career. You might call it transdisciplinary, because what I’m really looking at is not just computer science and linguistics and the kind of cognitive processes that underline language, but also now bringing in the molecular biology. I worked on protein function prediction for a while from the perspective of processing scientific language in journal papers. And from there, into the clinical data analysis, because realizing that all of these molecular things connect to the symptoms and the diseases that we see in the human body and the documentation practices of clinicians to try to capture all of that. So, I think you’re right. I couldn’t have done this if I hadn’t had that foundation of really thinking about the bigger picture and how everything is interconnected. And in fact, words are sort of the embodiment of that because our words don’t live in isolation when we speak. They’re all interconnected. And so, having this ability to kind of step back and think about how concepts and the way we talk about them relate to each other allows me to think more systematically about how things in general relate to each other and how diseases connect to genetic mutations and the chemicals that we use to treat them. So, yeah, it’s a really complex, interconnected world. And I think having that ability and exposure to multiple ways of thinking about these things really enabled me to be effective in that interdisciplinary space.

Miranda Prynne: Yeah, and AI, you know, for all the complexities and challenges that it does present, of course, is also a fantastic tool for accelerating that kind of transdisciplinary way of working, just because it can process so much more information from many different perspectives. Can you give us a potted history of how AI has developed, specifically as a biomedical research tool, in just the last 10 years or so?

Karin Verspoor: The last 10 years have been tremendous. We have seen a massive increase in, first of all, the amount of data that’s available publicly for us to be working with computers. But second of all, the complexity and sophistication of the machine learning models that have been developed has just jumped. It’s just really sort of leapfrogged. And the combination of availability of data, increases in compute power, and increases in the sophistication of our machine learning models has meant that we can just do things now that we couldn’t do 10 years ago. We’re able to really tap into the huge amounts of information that are embodied by all of that data and put those compute tools to work to really find the statistical patterns and make inferences about connections between things and find patterns in that data that the human eye or the human mind can’t manage to find just because there’s so much of it.

Miranda Prynne: Yes, and on the subject of data, a major and very well-documented risk of AI is that it reproduces the biases of its data input. So, how do you mitigate this? And I mean, can you sort of offer advice for other researchers on just how to approach that particular challenge?

Karin Verspoor: Yeah, it is a real challenge. And in fact, that ends up being something we have to study to understand exactly what the extent of the bias is. You know, sometimes bias is not a bad thing in the sense that it reflects the realities of the real world. You know, our world is not unbiased. And so, if a model is accurately capturing the patterns in the data, then actually it’s done the right thing. The problem is that we don’t want to exacerbate biases by having models that are biased in the wrong kind of context. And so, we have to be really careful to interrogate very closely what the models are doing under particular conditions. I always like to talk about the fact that, you know, something that’s really common will be really easy for these models to capture because the statistics just add up. And the things that are quite challenging for these models are things that are in, you know, what we think of in statistical terms, the tails of the distribution or more rare events. And so that means we need to be really careful once we move out of the kind of middle of the distribution where there’s a lot of certainty and a lot of redundancy of information to things which are a bit more rare, then we need to really worry about what the model is doing under those conditions. And having models that we just assume are always going to work no matter what context we deploy them in is naive. We need to ensure that we’re really looking at the context where a model is being deployed and adapt them to be robust to the changes in the underlying populations.

Miranda Prynne: So where there’s essentially a smaller sample size, you need to really interrogate how the model is judging what it does more closely.

Karin Verspoor: Yeah, that’s exactly right. So, you know, where we know there are biases in the world, we want to make sure that our models are doing the right things when we’re in that kind of smaller sample zone of the data. Maybe I’ll just give a little example. You know, when we think about rare diseases, for example, you might have one in 50,000 people with a particular disease. And we can’t expect that any hospital-sized data set would actually have enough information to give you any advice about what to do with that one person because hospitals only see a limited number of patients. And so they may have only seen two or three of these people over the course of, say, a 10-year period. And so we can’t rely on the existing data to help inform how to treat that patient. Now, on the other hand, if we can bring together data from more hospitals and at the national level or at the international level, then we have a hope of finding some patterns that are meaningful. But at that one hospital level, we can’t do that. And the reality is that most of the models we build today, at least in the clinical domain, are based on quite small data sets and quite small populations, typically only that one hospital level. And that means, you know, we have to be really aware that the models are not fit for purpose in rare disease conditions, typically.

Miranda Prynne: So, in general terms, the more data, the better. Always. That reflects research more generally, really.

Karin Verspoor: Absolutely, absolutely. And look, we often don’t have big data, right? We don’t always have the luxury of having those big data sets. You know, some of the things that we’re seeing now, like the protein structure prediction with AlphaFold, has the benefit of decades of capturing protein data in structured databases. And these models are effective precisely because they’ve seen huge amounts of data. But in many clinical applications, we just don’t have access to that kind of public data. And so this is a real challenge. And we need to be really careful about how we do this. And one challenge is that trying to get this data into the shape that it’s going to work for a machine learning algorithm to even try to build a model with the data requires a lot of human input. We need to interpret the data for the machine learning model, right? So basically, we need to provide examples of how to interpret particular data to a model before the model can try to interpret it itself. So this requires a lot of resources and it’s quite a challenge. But we’re working at the moment on some methods for taking small data sets and essentially trying to generalize what we can learn from a small data set to another small data set that’s closely related. This is a process that we call transfer learning, so that we can essentially abstract away from some of the details of a specific data set to a closely related but different data set in order to facilitate that generalization to a new kind of problem or a new population.

Miranda Prynne: Coming back to your research, because as you’ve explained, it focuses a lot on language linguistics. Language changes quite a lot over time. Even in the course of a decade, there will be quite a major sort of revolution of certain terms that come in and out of vogue or just become more common. So, do you have to build that into your AI tools to kind of recognize the changing ways people refer to things over the course of time?

Karin Verspoor: Luckily, in the work that we’re doing, we’re not as exposed to the kind of general language trends that are impacting, say, social media. There are people who work on social media analysis, even in the health context, and that can be an area where they really do need to be looking at that language evolution and the vernacular as it changes over time. In scientific contexts, in particular, obviously, we have new concepts that are being introduced, you know, new diseases being described or new genes being identified as particular genetic targets. But the changes are a lot more minor. The broader language around how we describe biological processes or disease mechanisms isn’t really changing that much. And so we can rely on kind of that certainty of, well, you know, if you’re talking about a protein-protein interaction, you’re probably going to be using particular terminology about how those proteins are interacting on a biological level and not expect to find too many words that are changing in that kind of context of setting up the relationships between them.

Miranda Prynne: Yeah, that makes sense and actually pretty makes the case for scientific language being what it is. Regina Barzilay, who I’m sure you’re aware of and who has developed an AI tool to support breast cancer screening, I read a piece from actually a couple of years ago in which she was calling for more transparency over AI use in medicine and healthcare to ensure fair representation in the training models and thus more accurate diagnoses across different populations. I wondered what your thoughts on this were and if you have observed any progress being made in that area.

Karin Verspoor: With any AI model, it’s really important that we understand how it was trained, how it was evaluated, you know, what the level of accuracy was over that, over the test set. And absolutely, we need to have transparency about where the data come from, precisely so that we can understand what potential errors it might make, what biases might be embedded in the model. And so, I agree with Regina entirely that we need to document and disclose how models were trained. In reality, at the moment, that is becoming more and more challenging, particularly with the large language models, of course, where they’re being trained on very large, very open data sets, and we don’t actually know exactly what they’ve been trained on. This also, by the way, becomes a problem for evaluating these models because they may contain the material that we’re trying to use to test the efficacy of the models. And so, essentially, all we’re doing is testing whether they’ve memorized the data that they’ve been exposed to rather than actually testing how they’ll work on new data that they haven’t seen before. So, knowing where the data comes from is a way of saying, like, I can be confident that the data that I’m using to test the model is different from the model that it’s been trained on. But it’s getting harder and harder to get that information.

Miranda Prynne: Yes, on that basis, do you think there is still a place for Gen AI and the LLMs in research, or should academics tread pretty carefully when using them?

Karin Verspoor: So, you know, these tools have some really, really clear advantages. We can use language to interact with these tools, right? Which puts us in a space where we don’t have to adapt the way we want to look for information to the tool, but the tool is actually sort of meeting us where we are because it’s using the same words and it’s able to respond to our language in a very natural way because the model is inherently based on language. And so, Gen AI tools give us a way to ask kind of open-ended questions in a really natural way that hopefully gets a meaningful response in return. Where we run into trouble is where we are relying on Gen AI to make inferences or decisions that we can’t scrutinize. And that’s because the models, kind of as I was saying earlier, they may not have seen the sorts of cases where we want to apply it. And there’s a lot of uncertainty about whether it’s going to give us the right response. And so I think we have to find a balance. You know, essentially, we want to make sure that we’re always trying to integrate whatever information we get back from an LLM into our understanding of the particular context or the particular problem we’re trying to address, because otherwise, there’s the potential for disconnect between essentially what the model has been exposed to and what we’re actually trying to do in a specific situation may be too big.

Miranda Prynne: Would it help if there were clearer standards from government funders, journals on the reporting of data sources and demographic breakdowns covering things like race, gender, age, and so on?

Karin Verspoor: Certainly, when we’re talking about supervised machine learning models, so AI models that have been developed for a particular purpose to solve a particular problem, where we’re providing them with a particular type of data and expecting some specific kind of output, like trying to predict an outcome for a particular patient with a particular disease that’s being given a particular treatment, we expect that the model will behave in a particular way. And we need to know how the model was trained so that we can detect when there’s kind of a mismatch between the data that we have about a particular patient and the data that was used to train and test the model. In the generative space, things become a lot more tricky because of that open-endedness. It’s not a specific task. It’s not supervised to solve a particular problem. It’s open-ended. We can ask whatever we want. And that’s where things get tricky. Documenting exactly what’s in there may or may not actually give us insight into how it’s going to perform because it’s statistical and therefore and generative and therefore unpredictable.

Miranda Prynne: Yes, and we know it will also fill in gaps itself where it can’t actually access the information.

Karin Verspoor: No, that’s right. And we need to ensure that the tools are fit for purpose. And that’s really the point, right? So if you want to use a Gen AI-based tool to help you do a literature review, for example, you want to have some confidence that you’re working with a tool that, first of all, understands what a literature review is and what the steps are in terms of ensuring the quality of that review, but also that it’s interactive so that you can guide it in the direction that’s meaningful for you. And then when we’re talking about tools that are more about supporting decisions, that’s where we want to have more assurance that it’s doing what we expect it to do.

Miranda Prynne: It does sound like the answer generally lies in more tailored AI tools for researchers if they want to be using them in more complex and nuanced ways for discovery.

Karin Verspoor: That’s certainly the direction my work is heading in. And I think, you know, we do have to find a balance because if you have a tailored tool, that means that it’s baking in certain assumptions about how to solve a problem, what the problem is. And, you know, we don’t want to completely cut off the creativity and the potential for innovation that a more generative model might have. But I think we need to find a balance because in science, we are working on the fringes of knowledge and we are working in that, you know, in the tails of the distribution. That’s the whole point of science. We’re trying to push things forward. And so we want to build on the massive amount of information that we have in science, which has been almost exclusively communicated through scientific publications. We can’t ignore that information. That’s inherent to how we do science. But we also need to be aware that as soon as we are dealing with something that’s kind of making, you know, statistical connections between things in undirected ways, that we might be in territory where it’s just making stuff up. But, you know, sometimes making stuff up is not a bad thing. I think that’s we sometimes call that innovation, right? But needs to be kind of treated with the caution that anything that’s a hypothesis rather than evidenced should be.

Miranda Prynne: Yes, and that’s of course where the role of the expert, the scientist, the researcher comes in. I’m sure many will be pleased to hear that they are not redundant in this process.

Karin Verspoor: 100%. We need scientists to keep this on track, right? And, you know, really, you know, my work has been to help scientists do their jobs more efficiently and more effectively. I am, you know, cautious about pure Gen AI. And certainly we need to be just always looking at the outputs of these tools with the critical lens and our understanding. Everybody, I think, has had that experience where they use a Gen AI tool and they’re like, oh, that’s not right. Or even if they’re given a link back to some source information, they go and look at the source and say, oh, right, the summary that I saw over in the AI tool isn’t quite the same as what I’m seeing in the raw material. So we need to do the proper kind of scientific process of checking our sources, making sure that it gels with our understanding of the domain, and really integrating the positives of these tools with our normal critical and scientific processes for just checking and verifying that the results are what we expect them to be.

Miranda Prynne: In summary, what, in your view, are the foundational principles of responsible, ethical AI use in research?

Karin Verspoor: I think in a nutshell, it’s use your brain, right? Remember that you’re an active participant in all of this. We should not be seeing these tools as taking over the scientific process, and that we just need to always think critically. There’s also that aspect of documenting how the tools that we’re using were developed and producing evaluations of how well they work and under what conditions we’ve tested them, because that’s also part of having confidence that these models will behave in new situations in expected ways.

Miranda Prynne: Yes, yeah, of course. Are there perhaps some things that should never be automated?

Karin Verspoor: Oh, that’s a tricky question.

Miranda Prynne: Yes, yeah, it is. I’m just interested if you think there are certain tasks that will always fall beyond the remit of the machines, or perhaps should always fall beyond their remit and require very much human, direct human handling.

Karin Verspoor: Everything needs human oversight. Actually, I think one thing that we really need to remember is that these tools can’t tell us what we value. And particularly in clinical applications, but also in other contexts, highly sensitive contexts, let’s say defense. We need to make sure that we’re using these tools in ways that still leave room for us to put in what matters to us. One example is an end-of-life decision, for example. You wouldn’t want an AI tool that says, I think this person should go into palliative care. And we don’t have any room for saying, well, is palliative care right for this particular person? And making decisions about whether to go home or whether to spend time with family or whether to use heroic measures in the hospital to try to save someone’s life or prolong someone’s life. That’s a very personal decision. And there’s no right or wrong answer there. And we don’t want an AI to make that decision. And that’s kind of an extreme example, perhaps, but in general, thinking about where do our human values and our individual values, which will vary and are not necessarily something that a machine can predict, we certainly don’t want to take over that kind of decision in the AI.

Miranda Prynne: Yeah, well put. Thinking about all of this, do you think we ultimately should be encouraging more researchers to engage with AI in their work and training academics accordingly?

Karin Verspoor: I think that researchers are using these tools anyway, right? The ship has sailed, the horse has bolted, whatever metaphor you want to use, it’s out there, people are using it. I don’t actually think we need to encourage them if they’re, you know, if they’re finding value in these tools, they will continue to use them. On the other hand, I do see a lot of value in leveraging the tools of AI to help us manage the information we have. And the computational models are really, really powerful. At the moment, we’re so nervous about what can go wrong with Gen AI in particular that we tend to shy away from the really positive things that we can do with these technologies. I mentioned earlier the rare disease situation. To solve that problem at a kind of international level of being able to get information to a patient with a rare disease about what a good treatment option might be, you really need a very, very large and powerful model that has access to a huge amount of information. And if we are always worried about the risks that we might introduce in that situation, we’re missing an opportunity to really leverage the power of the computation to find connections and patterns in the data that we just might not even be aware is there. So I think we have to find a balance between using the tools to help us and ensuring that we don’t introduce problems or errors into the process. We don’t want to throw the baby out with the bathwater.

Miranda Prynne: Yeah, of course. And AI has already underpinned massive advances across the board in research, the number of sort of healthcare applications and things that have been made achievable. So yes, of course that needs to be remembered amidst the widespread panic caused by Gen AI that maybe is calming slightly now.

Karin Verspoor: We can also say that people make mistakes too, right? So, you know, just because an AI system or a technology isn’t 100% perfect doesn’t mean it doesn’t have value. So we just, we need to balance the different aspects of this and try to use it where it’s helpful and hopefully avoid some of the pitfalls. It’s a tool like any other.

Miranda Prynne: 100%. Thank you so much for talking to me today, Karin. It’s been absolutely fascinating.

Karin Verspoor: You’re welcome. Thanks so much for having me.

Miranda Prynne: Karin’s position is pretty clear. These tools have huge potential to do good and we shouldn’t let our suspicions and worries over their imperfections impede this, but they need to be properly managed. They need to be critiqued and understood and there must be proper human oversight, which is quite tricky with current Gen AI models. Her straightforward message to simply use your brain is one that we should all take away from this. I think it’s becoming increasingly clear that we must not extradite human thinking, insight and judgment to machines. And really, we must remember that they are there to help and not replace.

Miranda Prynne: If you want more advice on the practicalities of using AI, in particular Gen AI, as a research assistant, we have a whole collection of resources exploring this topic with advice from experts all over the world. You will find all that at www.timeshighereducation.com/campus.

Miranda Prynne: If you enjoyed this podcast, please do like and subscribe, and we will be back with you in a couple of weeks. Thank you. Bye-bye.

Sound clip: You’re listening to a Times Higher Education podcast.