LLM

The LLM pipeline runs prompts through a large language model (LLM). This pipeline autodetects the LLM framework based on the model path.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import LLM

# Create and run LLM pipeline
llm = LLM()
llm(
  """
  Answer the following question using the provided context.

  Question:
  What are the applications of txtai?

  Context:
  txtai is an open-source platform for semantic search and
  workflows powered by language models.
  """
)

The LLM pipeline automatically detects the underlying LLM framework. This can also be manually set.

from txtai.pipeline import LLM

# Set method as litellm
llm = LLM("vllm/Open-Orca/Mistral-7B-OpenOrca", method="litellm")

# Set method as llama.cpp
llm = LLM("TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
           method="llama.cpp")

Models can be externally loaded and passed to pipelines. This is useful for models that are not yet supported by Transformers and/or need special initialization.

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from txtai.pipeline import LLM

# Load Mistral-7B-OpenOrca
path = "Open-Orca/Mistral-7B-OpenOrca"
model = AutoModelForCausalLM.from_pretrained(
  path,
  torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(path)

llm = LLM((model, tokenizer))

See the links below for more detailed examples.

Notebook	Description
Prompt-driven search with LLMs	Embeddings-guided and Prompt-driven search with Large Language Models (LLMs)
Prompt templates and task chains	Build model prompts and connect tasks together with workflows
Build RAG pipelines with txtai	Guide on retrieval augmented generation including how to create citations
Integrate LLM frameworks	Integrate llama.cpp, LiteLLM and custom generation frameworks
Generate knowledge with Semantic Graphs and RAG	Knowledge exploration and discovery with Semantic Graphs and RAG
Build knowledge graphs with LLMs	Build knowledge graphs with LLM-driven entity extraction
Advanced RAG with graph path traversal	Graph path traversal to collect complex sets of data for advanced RAG
Advanced RAG with guided generation	Retrieval Augmented and Guided Generation

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

# Create pipeline using lower case class name
# Use `generator` or `sequences` to force model type
llm:

# Run pipeline with workflow
workflow:
  llm:
    tasks:
      - action: llm

Similar to the Python example above, the underlying Hugging Face pipeline parameters and model parameters can be set in pipeline configuration.

llm:
  path: Open-Orca/Mistral-7B-OpenOrca
  torch_dtype: torch.bfloat16

Run with Workflows

from txtai.app import Application

# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("llm", [
  """
  Answer the following question using the provided context.

  Question:
  What are the applications of txtai? 

  Context:
  txtai is an open-source platform for semantic search and
  workflows powered by language models.
  """
]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &

curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"sequences", "elements": ["Answer the following question..."]}'

Methods

Python documentation for the pipeline.

Creates a new LLM.

Parameters:

Name	Type	Description	Default
path		model path	None
method		llm model framework, infers from path if not provided	None
kwargs		model keyword arguments	{}

Source code in txtai/pipeline/llm/llm.py

def __init__(self, path=None, method=None, **kwargs):
    """
    Creates a new LLM.

    Args:
        path: model path
        method: llm model framework, infers from path if not provided
        kwargs: model keyword arguments
    """

    # Default LLM if not provided
    path = path if path else "google/flan-t5-base"

    # Generation instance
    self.generator = GenerationFactory.create(path, method, **kwargs)

Generates text using input text

Parameters:

Name	Type	Description	Default
text		text|list	required
maxlength		maximum sequence length	512
kwargs		additional generation keyword arguments	{}

Returns:

Type	Description
	generated text

Source code in txtai/pipeline/llm/llm.py

def __call__(self, text, maxlength=512, **kwargs):
    """
    Generates text using input text

    Args:
        text: text|list
        maxlength: maximum sequence length
        kwargs: additional generation keyword arguments

    Returns:
        generated text
    """

    # Run LLM generation
    return self.generator(text, maxlength, **kwargs)