FAQ

Below is a list of frequently asked questions and common issues encountered.

Questions

---

Question

What models are recommended?

Answer

See the model guide.

---

Question

What is the best way to track the progress of an embeddings.index call?

Answer

Wrap the list or generator passed to the index call with tqdm. See #478 for more.

---

Question

What is the best way to analyze the content of a txtai index?

Answer

txtai has a console application that makes this easy. Read this article to learn more.

---

Question

How can models be externally loaded and passed to embeddings and pipelines?

Answer

Embeddings example.

from transformers import AutoModel, AutoTokenizer
from txtai.embeddings import Embeddings

# Load model externally
model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

# Pass to embeddings instance
embeddings = Embeddings(path=model, tokenizer=tokenizer)

LLM pipeline example.

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from txtai.pipeline import LLM

# Load Mistral-7B-OpenOrca
path = "Open-Orca/Mistral-7B-OpenOrca"
model = AutoModelForCausalLM.from_pretrained(
  path,
  torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(path)

llm = LLM((model, tokenizer))

Common issues

---

Issue

Embeddings query errors like this:

SQLError: no such function: json_extract

Solution

Upgrade Python version as it doesn't have SQLite support for json_extract

---

Issue

Segmentation faults and similar errors on macOS

Solution

Disable OpenMP threading via the environment variable export OMP_NUM_THREADS=1 or downgrade PyTorch to <= 1.12. See issue #377 for more.

---

Issue

ContextualVersionConflict and/or package METADATA exception while running one of the examples notebooks on Google Colab

Solution

Restart the kernel. See issue #409 for more on this issue.

---

Issue

Error installing optional/extra dependencies such as pipeline

Solution

The default MacOS shell (zsh) and Windows PowerShell require escaping square brackets

pip install 'txtai[pipeline]'