What does it mean for information seekers then? Should they not trust or use LLM-based systems at all? I believe we want to trust and use these systems, but we should not do it blindly. As Reagan said, “Trust, but verify.” So how do we do that? I have three suggestions.

First, user education. Many users don’t quite understand how generative AI tools operate and what their limitations are. When one sees a chatbot doing human-like conversations and providing what seems like a pretty good answer or recommendation, it’s easy to miss the fact that these are not models with intents, empathy, or even real knowledge. They have wonderfully captured relations among the billions of tokens fed into them, and are able to spit out a sequence of tokens in a stochastic manner, giving us an impression of comprehension, but they are not understanding or generating things the way humans do. They are certainly not all-knowing agents as many expect. If an average user could understand this, I believe they could start to see and use these tools for what they are—tools for information extraction and generation, but nothing more. They don’t feel or think. They also don’t know everything. And that’s fine. They don’t have to, but at least now when we get some response from them, we can take it with a grain of salt and even question or validate things on our own. That simple education of what the LLMs are and what they are not can go a long way to establish the right kind of trust between them and the users.

Second, transparency. This goes with the previous point, but it’s so important that it needs to be stated explicitly. Take the seizure example from above. The system tells us that the answer is taken from a reputable healthcare site. That’s a form of transparency and a welcome step, but we already saw why it is not enough. A real transparent system will not only tell us where the answers are coming from, but also inform how they are extracted or generated. Several LLM-based search services have started giving citations to indicate the sources from where the answer was generated. But if you ever check these sources, you may find that many times you can’t find the exact or even close enough text from that answer. That’s not useful kind of transparency. Worse, because a user sees the service providing such citations, they may be misled to believe the answer must be correct or authoritative—much like the seizure example above. We need better transparency so that the users can question the answers they should be questioning and accept the ones they should—building the right kind of trust.  

Third, innovations in building trustworthy systems. At the end of the day, the burden for an LLM or an LLM-driven system to be trustworthy shouldn’t be on the end-user, who may lack motivation or skills to verify everything. There are innovations to be made here by system designers and developers. Some of the current approaches include retrieval-augmented generation (RAG) and evidence attribution. These approaches either do generation from already verified content or use a trustworthy third party to verify what an LLM generates. There are limitations to these and even in the best case, these approaches don’t work for all situations. We need other, more effective ways to do validation and attribution.