LlamaIndex - 2편 (QA 및 평가)

생산 레벨의 예

SEC-Insights

QA

User Case:

• Q&A: 很不错
• Structured Data Extraction

What

• 의미 쿼리(Semantic search / Top K)
• 요약

Where

• Over documents
• Building a multi-document agent over the LlamaIndex docs
• Over structured data(예: JSON)
• Searching Pandas tables
• Text to SQL

How

• Combine multiple sources
• Route across multiple sources
• Multi-document queries

위의 링크는 모두 아래 Q&A patterns를 가리킵니다.

Understanding: Q&A patterns

가장 간단한 Q&A

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)

다른 데이터 소스 선택

链接

Compare/Contrast Queries

이건 잘 모르겠어

Multi Document Queries

Besides the explicit synthesis/routing flows described above, LlamaIndex can support more general multi-document queries as well.SubQuestionQueryEngine class.Given a query, this query engine will generate a 'query plan' containing sub-queries against sub-documents before synthesizing the final answer.

This query engine can execute any number of sub-queries against any subset of query engine tools before synthesizing the final answer.This makes it especially well-suited for compare/contrast queries across documents as well as queries pertaining to a specific document.

Multi-Step Queries

LlamaIndex can also support iterative multi-step queries.Given a complex query, break it down into an initial subquestions, and sequentially generate subquestions based on returned answers until the final answer is returned.

"Who was in the first batch of the accelerator program the author started?"

时态查询

에발

概念入门

• 응답 평가
• 평가 검색

详解概述和流程

• 응답 평가
• GPT-4를 사용하여 평가
• 평가 차원
• 생성된 답변 및 참조 답변: 정확성 및 의미 유사성
• 생성된 답변 및 retrieved contexts:Faithfulness
• 쿼리에 대한 결과 답변: Answer Relevancy
• retrieved contexts 및 Query:Context Relevancy
• 참조 응답 생성
• 평가 검색(retrieval)
• 평가 방법: MRR(ranking metrics like mean-reciprocal rank), hit-rate, precision, and more.

生成dataset

샘플 사용

• Query Eval Usage Pattern
• Retrieval Eval Usage Pattern

다른 도구에 통합

• UpTrain: 1.9K：可试用，但是需要book demo，目测不便宜
• Tonic Validate(Includes Web UI for visualizing results)：有商业版本，可试用，之后200美元/月
  • llamaindex有详细文章
• DeepEval: 1.6K
• Ragas: 4.4K
• 괜찮은 것 같아요
• Llamaindex-->Ragas-->LangSmith 및 기타 도구
• 하지만, quick start가 실패하여 ModuleNotFoundError: No module named 'ragas.metrics'; 'ragas' is not a package라는 메시지가 함께 표시됩니다.

费用评估

최적화

基础优化

Retrieval