Mismatch Between Query Response and Source Document Ordering in a Captive Network with Multiple Documents about h2ogpt HOT 7 OPEN

Smarter models do better at such tasks.

https://github.com/h2oai/enterprise-h2ogpte/blob/main/rag_benchmark/results/test_client_e2e.md

from h2ogpt.

If you can give me an example repro of your case using some set of public documents, where it shows sources are at bottom of list, I can take a look.

The sources are scored based upon standard semantic similarity search and the distance metric 'l2'. The LLM then decides what is most relevant from that. As you might imagine, that's why LLMs are so useful, is because they actually find the relevant info, when semantic similarity cannot fully. That's why we fill the context with various info the let the LLM decide.

There's no easy way to extract which text/sources the LLM used, that's an area of active research. In some cases a simple ngram check may help, but it also might hurt for non-keyword related cases. There may be a way to back-prop through the model to see why it decided, but nobody has figured that out for auto-regressive case, only classification.

If the user has to verify the answer, best is to present the sources more cleanly like in our Enterprise h2oGPT https://h2o.ai/resources/video/enterprise-h2ogpt/ that shows full page references with highlighted text regions. But that doesn't really solve your problem when there are numerous documents.

from h2ogpt.

how does one get the highlighted text in the source when one clicks in the opensource version?

from h2ogpt.

There's no highlighted text in OSS version.

from h2ogpt.

How does one resolve the "detail": "File not allowed" error , when one clicks on the sources?

from h2ogpt.

^ #1512

from h2ogpt.

A further question: the LLM is unable to provide the information from the document which is given in a Table in the PDF. Even after asking specifically which table to see it is unable to give the results.

For example if one wants to know what all are the company's investments it is unable to list it down (See table 3B in the 2023 pdf).

Going forward, aim was to find out which new investments happened during the year which could be done by comparing annual reports of 2 years, the LLM is unable to to that even after pointing which 2 tables to compare.

Annual Reports: https://www.infoedge.in/InvestorRelations/IR_Annual_Report

from h2ogpt.