python-3.x langchain中多个csv文件的问答

r6vfmomb  于 2023-04-22  发布在  Python
关注(0)|答案(1)|浏览(504)

我有一个包含多个csv文件的文件夹,我正在试图找出一种方法来将它们全部加载到langchain中,并对它们进行提问。
这是我目前掌握的情况

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain import OpenAI, VectorDBQA
from langchain.document_loaders import DirectoryLoader
from langchain.document_loaders.csv_loader import CSVLoader
import magic
import os
import nltk

os.environ['OPENAI_API_KEY'] = '...'

loader = DirectoryLoader('../data/', glob='**/*.csv', loader_cls=CSVLoader)

documents = loader.load()

text_splitter = CharacterTextSplitter(chunk_size=400, chunk_overlap=0)

texts = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings(openai_api_key=os.environ['OPENAI_API_KEY'])

docsearch = Chroma.from_documents(texts, embeddings)

qa = VectorDBQA.from_chain_type(llm=OpenAI(), chain_type="stuff", vectorstore=docsearch)

query = "how many females are present?"
qa.run(query)
xjreopfe

xjreopfe1#

你应该把它们都加载到一个vectorstore中,比如PineconeMetal。然后根据你是否需要内存,使用RetrievalQA链或ConversationRetrievalChain

相关问题