LLM stability

We are finding LLM to be somewhat unstable and inconsistent when we query LLM’s asking for summaries of recent research. We have found Claude and scite best for accurate and meaningful scientific reviews.

The LLM will often reply saying that they do not have access to the paper requested, and then continue on to give an accurate review of that paper. At other times it will continue but with a generic answer - not exactly a hallucination but the machine is trying hard to reply without all the information it would want.

Another answer may tell us that the machine does not have access to papers from the last few years, and then continue to give an answer referencing a paper from 2022. Sometimes it says that it cannot read attachments, or cannot access links, and then, seemingly proceed to do the same.

Queries that are answered on one day are bounced on another.

Perhaps it is because we are pushing the machines to work on texts that have only just been published. On the one hand the machines may be ingesting recent research papers, but the algorithm may not be willing to admit it. Perhaps recently ingested material may take a while to be assimilated into the machines stable knowledge.

Equally if we give a link to an article, the answer may tell us that it cannot follow a link and then behave as if it has.

These LLM’s are amazingly accurate when they oblige, but we are interested to see if these inconsistencies will stabilize.