Skip to content

Methods

Embeddings databases are the engine that delivers semantic search. Data is transformed into embeddings vectors where similar concepts will produce similar vectors. Indexes both large and small are built with these vectors. The indexes are used to find results that have the same meaning, not necessarily the same keywords.

Source code in txtai/embeddings/base.py
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 412
 413
 414
 415
 416
 417
 418
 419
 420
 421
 422
 423
 424
 425
 426
 427
 428
 429
 430
 431
 432
 433
 434
 435
 436
 437
 438
 439
 440
 441
 442
 443
 444
 445
 446
 447
 448
 449
 450
 451
 452
 453
 454
 455
 456
 457
 458
 459
 460
 461
 462
 463
 464
 465
 466
 467
 468
 469
 470
 471
 472
 473
 474
 475
 476
 477
 478
 479
 480
 481
 482
 483
 484
 485
 486
 487
 488
 489
 490
 491
 492
 493
 494
 495
 496
 497
 498
 499
 500
 501
 502
 503
 504
 505
 506
 507
 508
 509
 510
 511
 512
 513
 514
 515
 516
 517
 518
 519
 520
 521
 522
 523
 524
 525
 526
 527
 528
 529
 530
 531
 532
 533
 534
 535
 536
 537
 538
 539
 540
 541
 542
 543
 544
 545
 546
 547
 548
 549
 550
 551
 552
 553
 554
 555
 556
 557
 558
 559
 560
 561
 562
 563
 564
 565
 566
 567
 568
 569
 570
 571
 572
 573
 574
 575
 576
 577
 578
 579
 580
 581
 582
 583
 584
 585
 586
 587
 588
 589
 590
 591
 592
 593
 594
 595
 596
 597
 598
 599
 600
 601
 602
 603
 604
 605
 606
 607
 608
 609
 610
 611
 612
 613
 614
 615
 616
 617
 618
 619
 620
 621
 622
 623
 624
 625
 626
 627
 628
 629
 630
 631
 632
 633
 634
 635
 636
 637
 638
 639
 640
 641
 642
 643
 644
 645
 646
 647
 648
 649
 650
 651
 652
 653
 654
 655
 656
 657
 658
 659
 660
 661
 662
 663
 664
 665
 666
 667
 668
 669
 670
 671
 672
 673
 674
 675
 676
 677
 678
 679
 680
 681
 682
 683
 684
 685
 686
 687
 688
 689
 690
 691
 692
 693
 694
 695
 696
 697
 698
 699
 700
 701
 702
 703
 704
 705
 706
 707
 708
 709
 710
 711
 712
 713
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
class Embeddings:
    """
    Embeddings databases are the engine that delivers semantic search. Data is transformed into embeddings vectors where similar concepts
    will produce similar vectors. Indexes both large and small are built with these vectors. The indexes are used to find results
    that have the same meaning, not necessarily the same keywords.
    """

    # pylint: disable = W0231
    def __init__(self, config=None, models=None, **kwargs):
        """
        Creates a new embeddings index. Embeddings indexes are thread-safe for read operations but writes must be synchronized.

        Args:
            config: embeddings configuration
            models: models cache, used for model sharing between embeddings
            kwargs: additional configuration as keyword args
        """

        # Index configuration
        self.config = None

        # Dimensionality reduction - word vectors only
        self.reducer = None

        # Dense vector model - transforms data into similarity vectors
        self.model = None

        # Approximate nearest neighbor index
        self.ann = None

        # Index ids when content is disabled
        self.ids = None

        # Document database
        self.database = None

        # Resolvable functions
        self.functions = None

        # Graph network
        self.graph = None

        # Sparse vectors
        self.scoring = None

        # Query model
        self.query = None

        # Index archive
        self.archive = None

        # Subindexes for this embeddings instance
        self.indexes = None

        # Models cache
        self.models = models

        # Merge configuration into single dictionary
        config = {**config, **kwargs} if config and kwargs else kwargs if kwargs else config

        # Set initial configuration
        self.configure(config)

    def score(self, documents):
        """
        Builds a term weighting scoring index. Only used by word vectors models.

        Args:
            documents: iterable of (id, data, tags), (id, data) or data
        """

        # Build scoring index for word vectors term weighting
        if self.isweighted():
            self.scoring.index(Stream(self)(documents))

    def index(self, documents, reindex=False):
        """
        Builds an embeddings index. This method overwrites an existing index.

        Args:
            documents: iterable of (id, data, tags), (id, data) or data
            reindex: if this is a reindex operation in which case database creation is skipped, defaults to False
        """

        # Initialize index
        self.initindex(reindex)

        # Create transform and stream
        transform = Transform(self, Action.REINDEX if reindex else Action.INDEX)
        stream = Stream(self, Action.REINDEX if reindex else Action.INDEX)

        with tempfile.NamedTemporaryFile(mode="wb", suffix=".npy") as buffer:
            # Load documents into database and transform to vectors
            ids, dimensions, embeddings = transform(stream(documents), buffer)
            if embeddings is not None:
                # Build LSA model (if enabled). Remove principal components from embeddings.
                if self.config.get("pca"):
                    self.reducer = Reducer(embeddings, self.config["pca"])
                    self.reducer(embeddings)

                # Save index dimensions
                self.config["dimensions"] = dimensions

                # Create approximate nearest neighbor index
                self.ann = self.createann()

                # Add embeddings to the index
                self.ann.index(embeddings)

            # Save indexids-ids mapping for indexes with no database, except when this is a reindex
            if ids and not reindex and not self.database:
                self.ids = self.createids(ids)

        # Index scoring, if necessary
        # This must occur before graph index in order to be available to the graph
        if self.issparse():
            self.scoring.index()

        # Index subindexes, if necessary
        if self.indexes:
            self.indexes.index()

        # Index graph, if necessary
        if self.graph:
            self.graph.index(Search(self, True), Ids(self), self.batchsimilarity)

    def upsert(self, documents):
        """
        Runs an embeddings upsert operation. If the index exists, new data is
        appended to the index, existing data is updated. If the index doesn't exist,
        this method runs a standard index operation.

        Args:
            documents: iterable of (id, data, tags), (id, data) or data
        """

        # Run standard insert if index doesn't exist or it has no records
        if not self.count():
            self.index(documents)
            return

        # Create transform and stream
        transform = Transform(self, Action.UPSERT)
        stream = Stream(self, Action.UPSERT)

        with tempfile.NamedTemporaryFile(mode="wb", suffix=".npy") as buffer:
            # Load documents into database and transform to vectors
            ids, _, embeddings = transform(stream(documents), buffer)
            if embeddings is not None:
                # Remove principal components from embeddings, if necessary
                if self.reducer:
                    self.reducer(embeddings)

                # Append embeddings to the index
                self.ann.append(embeddings)

            # Save indexids-ids mapping for indexes with no database
            if ids and not self.database:
                self.ids = self.createids(self.ids + ids)

        # Scoring upsert, if necessary
        # This must occur before graph upsert in order to be available to the graph
        if self.issparse():
            self.scoring.upsert()

        # Subindexes upsert, if necessary
        if self.indexes:
            self.indexes.upsert()

        # Graph upsert, if necessary
        if self.graph:
            self.graph.upsert(Search(self, True), Ids(self), self.batchsimilarity)

    def delete(self, ids):
        """
        Deletes from an embeddings index. Returns list of ids deleted.

        Args:
            ids: list of ids to delete

        Returns:
            list of ids deleted
        """

        # List of internal indices for each candidate id to delete
        indices = []

        # List of deleted ids
        deletes = []

        if self.database:
            # Retrieve indexid-id mappings from database
            ids = self.database.ids(ids)

            # Parse out indices and ids to delete
            indices = [i for i, _ in ids]
            deletes = sorted(set(uid for _, uid in ids))

            # Delete ids from database
            self.database.delete(deletes)
        elif self.ann or self.scoring:
            # Find existing ids
            for uid in ids:
                indices.extend([index for index, value in enumerate(self.ids) if uid == value])

            # Clear embeddings ids
            for index in indices:
                deletes.append(self.ids[index])
                self.ids[index] = None

        # Delete indices for all indexes and data stores
        if indices:
            # Delete ids from ann
            if self.isdense():
                self.ann.delete(indices)

            # Delete ids from scoring
            if self.issparse():
                self.scoring.delete(indices)

            # Delete ids from subindexes
            if self.indexes:
                self.indexes.delete(indices)

            # Delete ids from graph
            if self.graph:
                self.graph.delete(indices)

        return deletes

    def reindex(self, config=None, function=None, **kwargs):
        """
        Recreates embeddings index using config. This method only works if document content storage is enabled.

        Args:
            config: new config
            function: optional function to prepare content for indexing
            kwargs: additional configuration as keyword args
        """

        if self.database:
            # Merge configuration into single dictionary
            config = {**config, **kwargs} if config and kwargs else config if config else kwargs

            # Keep content and objects parameters to ensure database is preserved
            config["content"] = self.config["content"]
            if "objects" in self.config:
                config["objects"] = self.config["objects"]

            # Reset configuration
            self.configure(config)

            # Reset function references
            if self.functions:
                self.functions.reset()

            # Reindex
            if function:
                self.index(function(self.database.reindex(self.config)), True)
            else:
                self.index(self.database.reindex(self.config), True)

    def transform(self, document):
        """
        Transforms document into an embeddings vector.

        Args:
            documents: iterable of (id, data, tags), (id, data) or data

        Returns:
            embeddings vector
        """

        return self.batchtransform([document])[0]

    def batchtransform(self, documents, category=None):
        """
        Transforms documents into embeddings vectors.

        Args:
            documents: iterable of (id, data, tags), (id, data) or data
            category: category for instruction-based embeddings

        Returns:
            embeddings vectors
        """

        # Initialize default parameters, if necessary
        self.defaults()

        # Default vector model, if necessary
        model = self.model if self.model else next(iter(self.models.values()))

        # Convert documents into sentence embeddings
        embeddings = model.batchtransform(Stream(self)(documents), category)

        # Reduce the dimensionality of the embeddings. Scale the embeddings using this
        # model to reduce the noise of common but less relevant terms.
        if self.reducer:
            self.reducer(embeddings)

        return embeddings

    def count(self):
        """
        Total number of elements in this embeddings index.

        Returns:
            number of elements in this embeddings index
        """

        if self.ann:
            return self.ann.count()
        if self.scoring:
            return self.scoring.count()
        if self.database:
            return self.database.count()
        if self.ids:
            return len([uid for uid in self.ids if uid is not None])

        # Default to 0 when no suitable method found
        return 0

    def search(self, query, limit=None, weights=None, index=None, parameters=None, graph=False):
        """
        Finds documents most similar to the input query. This method will run either an index search
        or an index + database search depending on if a database is available.

        Args:
            query: input query
            limit: maximum results
            weights: hybrid score weights, if applicable
            index: index name, if applicable
            parameters: dict of named parameters to bind to placeholders
            graph: return graph results if True

        Returns:
            list of (id, score) for index search
            list of dict for an index + database search
            graph when graph is set to True
        """

        results = self.batchsearch([query], limit, weights, index, [parameters], graph)
        return results[0] if results else results

    def batchsearch(self, queries, limit=None, weights=None, index=None, parameters=None, graph=False):
        """
        Finds documents most similar to the input queries. This method will run either an index search
        or an index + database search depending on if a database is available.

        Args:
            queries: input queries
            limit: maximum results
            weights: hybrid score weights, if applicable
            index: index name, if applicable
            parameters: list of dicts of named parameters to bind to placeholders
            graph: return graph results if True

        Returns:
            list of (id, score) per query for index search
            list of dict per query for an index + database search
            list of graph per query when graph is set to True
        """

        # Determine if graphs should be returned
        graph = graph if self.graph else False

        # Execute search
        results = Search(self, graph)(queries, limit, weights, index, parameters)

        # Create subgraphs using results, if necessary
        return [self.graph.filter(x) for x in results] if graph else results

    def similarity(self, query, data):
        """
        Computes the similarity between query and list of data. Returns a list of
        (id, score) sorted by highest score, where id is the index in data.

        Args:
            query: input query
            data: list of data

        Returns:
            list of (id, score)
        """

        return self.batchsimilarity([query], data)[0]

    def batchsimilarity(self, queries, data):
        """
        Computes the similarity between list of queries and list of data. Returns a list
        of (id, score) sorted by highest score per query, where id is the index in data.

        Args:
            queries: input queries
            data: list of data

        Returns:
            list of (id, score) per query
        """

        # Convert queries to embedding vectors
        queries = self.batchtransform(((None, query, None) for query in queries), "query")
        data = self.batchtransform(((None, row, None) for row in data), "data")

        # Dot product on normalized vectors is equal to cosine similarity
        scores = np.dot(queries, data.T).tolist()

        # Add index and sort desc based on score
        return [sorted(enumerate(score), key=lambda x: x[1], reverse=True) for score in scores]

    def explain(self, query, texts=None, limit=None):
        """
        Explains the importance of each input token in text for a query. This method requires either content to be enabled
        or texts to be provided.

        Args:
            query: input query
            texts: optional list of (text|list of tokens), otherwise runs search query
            limit: optional limit if texts is None

        Returns:
            list of dict per input text where a higher token scores represents higher importance relative to the query
        """

        results = self.batchexplain([query], texts, limit)
        return results[0] if results else results

    def batchexplain(self, queries, texts=None, limit=None):
        """
        Explains the importance of each input token in text for a list of queries. This method requires either content to be enabled
        or texts to be provided.

        Args:
            queries: input queries
            texts: optional list of (text|list of tokens), otherwise runs search queries
            limit: optional limit if texts is None

        Returns:
            list of dict per input text per query where a higher token scores represents higher importance relative to the query
        """

        return Explain(self)(queries, texts, limit)

    def terms(self, query):
        """
        Extracts keyword terms from a query.

        Args:
            query: input query

        Returns:
            query reduced down to keyword terms
        """

        return self.batchterms([query])[0]

    def batchterms(self, queries):
        """
        Extracts keyword terms from a list of queries.

        Args:
            queries: list of queries

        Returns:
            list of queries reduced down to keyword term strings
        """

        return Terms(self)(queries)

    def exists(self, path=None, cloud=None, **kwargs):
        """
        Checks if an index exists at path.

        Args:
            path: input path
            cloud: cloud storage configuration
            kwargs: additional configuration as keyword args

        Returns:
            True if index exists, False otherwise
        """

        # Check if this exists in a cloud instance
        cloud = self.createcloud(cloud=cloud, **kwargs)
        if cloud:
            return cloud.exists(path)

        # Check if this is an archive file and exists
        path, apath = self.checkarchive(path)
        if apath:
            return os.path.exists(apath)

        # Return true if path has a config.json or config file with an offset set
        return path and (os.path.exists(f"{path}/config.json") or os.path.exists(f"{path}/config")) and "offset" in self.loadconfig(path)

    def load(self, path=None, cloud=None, config=None, **kwargs):
        """
        Loads an existing index from path.

        Args:
            path: input path
            cloud: cloud storage configuration
            config: configuration overrides
            kwargs: additional configuration as keyword args
        """

        # Load from cloud, if configured
        cloud = self.createcloud(cloud=cloud, **kwargs)
        if cloud:
            path = cloud.load(path)

        # Check if this is an archive file and extract
        path, apath = self.checkarchive(path)
        if apath:
            self.archive.load(apath)

        # Load index configuration
        self.config = self.loadconfig(path)

        # Apply config overrides
        self.config = {**self.config, **config} if config else self.config

        # Approximate nearest neighbor index - stores dense vectors
        self.ann = self.createann()
        if self.ann:
            self.ann.load(f"{path}/embeddings")

        # Dimensionality reduction model - word vectors only
        if self.config.get("pca"):
            self.reducer = Reducer()
            self.reducer.load(f"{path}/lsa")

        # Index ids when content is disabled
        self.ids = self.createids()
        if self.ids:
            self.ids.load(f"{path}/ids")

        # Document database - stores document content
        self.database = self.createdatabase()
        if self.database:
            self.database.load(f"{path}/documents")

        # Sparse vectors - stores term sparse arrays
        self.scoring = self.createscoring()
        if self.scoring:
            self.scoring.load(f"{path}/scoring")

        # Subindexes
        self.indexes = self.createindexes()
        if self.indexes:
            self.indexes.load(f"{path}/indexes")

        # Graph network - stores relationships
        self.graph = self.creategraph()
        if self.graph:
            self.graph.load(f"{path}/graph")

        # Dense vectors - transforms data to embeddings vectors
        self.model = self.loadvectors()

        # Query model
        self.query = self.loadquery()

    def save(self, path, cloud=None, **kwargs):
        """
        Saves an index in a directory at path unless path ends with tar.gz, tar.bz2, tar.xz or zip.
        In those cases, the index is stored as a compressed file.

        Args:
            path: output path
            cloud: cloud storage configuration
            kwargs: additional configuration as keyword args
        """

        if self.config:
            # Check if this is an archive file
            path, apath = self.checkarchive(path)

            # Create output directory, if necessary
            os.makedirs(path, exist_ok=True)

            # Copy sentence vectors model
            if self.config.get("storevectors"):
                shutil.copyfile(self.config["path"], os.path.join(path, os.path.basename(self.config["path"])))

                self.config["path"] = os.path.basename(self.config["path"])

            # Save index configuration
            self.saveconfig(path)

            # Save approximate nearest neighbor index
            if self.ann:
                self.ann.save(f"{path}/embeddings")

            # Save dimensionality reduction model (word vectors only)
            if self.reducer:
                self.reducer.save(f"{path}/lsa")

            # Save index ids
            if self.ids:
                self.ids.save(f"{path}/ids")

            # Save document database
            if self.database:
                self.database.save(f"{path}/documents")

            # Save scoring index
            if self.scoring:
                self.scoring.save(f"{path}/scoring")

            # Save subindexes
            if self.indexes:
                self.indexes.save(f"{path}/indexes")

            # Save graph
            if self.graph:
                self.graph.save(f"{path}/graph")

            # If this is an archive, save it
            if apath:
                self.archive.save(apath)

            # Save to cloud, if configured
            cloud = self.createcloud(cloud=cloud, **kwargs)
            if cloud:
                cloud.save(apath if apath else path)

    def close(self):
        """
        Closes this embeddings index and frees all resources.
        """

        self.ann, self.config, self.graph, self.archive = None, None, None, None
        self.reducer, self.query, self.model, self.models = None, None, None, None
        self.ids = None

        # Close database connection if open
        if self.database:
            self.database.close()
            self.database, self.functions = None, None

        # Close scoring instance if open
        if self.scoring:
            self.scoring.close()
            self.scoring = None

        # Close indexes if open
        if self.indexes:
            self.indexes.close()
            self.indexes = None

    def info(self):
        """
        Prints the current embeddings index configuration.
        """

        if self.config:
            # Print configuration
            print(json.dumps(self.config, sort_keys=True, default=str, indent=2))

    def issparse(self):
        """
        Checks if this instance has an associated scoring instance with term indexing enabled.

        Returns:
            True if term index is enabled, False otherwise
        """

        return self.scoring and self.scoring.hasterms()

    def isdense(self):
        """
        Checks if this instance has an associated ANN instance.

        Returns:
            True if this instance has an associated ANN, False otherwise
        """

        return self.ann is not None

    def isweighted(self):
        """
        Checks if this instance has an associated scoring instance with term weighting enabled.

        Returns:
            True if term weighting is enabled, False otherwise
        """

        return self.scoring and not self.scoring.hasterms()

    def configure(self, config):
        """
        Sets the configuration for this embeddings index and loads config-driven models.

        Args:
            config: embeddings configuration
        """

        # Configuration
        self.config = config

        # Dimensionality reduction model
        self.reducer = None

        # Create scoring instance for word vectors term weighting
        scoring = self.config.get("scoring") if self.config else None
        self.scoring = self.createscoring() if scoring and (not isinstance(scoring, dict) or not scoring.get("terms")) else None

        # Dense vectors - transforms data to embeddings vectors
        self.model = self.loadvectors() if self.config else None

        # Query model
        self.query = self.loadquery() if self.config else None

    def initindex(self, reindex):
        """
        Initialize new index.

        Args:
            reindex: if this is a reindex operation in which case database creation is skipped, defaults to False
        """

        # Initialize default parameters, if necessary
        self.defaults()

        # Initialize index ids, only created when content is disabled
        self.ids = None

        # Create document database, if necessary
        if not reindex:
            self.database = self.createdatabase()

            # Reset archive since this is a new index
            self.archive = None

        # Initialize ANN, will be created after index transformations complete
        self.ann = None

        # Create scoring only if term indexing is enabled
        scoring = self.config.get("scoring")
        if scoring and isinstance(scoring, dict) and self.config["scoring"].get("terms"):
            self.scoring = self.createscoring()

        # Create subindexes, if necessary
        self.indexes = self.createindexes()

        # Create graph, if necessary
        self.graph = self.creategraph()

    def defaults(self):
        """
        Apply default parameters to current configuration.

        Returns:
            configuration with default parameters set
        """

        self.config = self.config if self.config else {}

        # Expand sparse index shortcuts
        if not self.config.get("scoring") and any(self.config.get(key) for key in ["keyword", "hybrid"]):
            self.config["scoring"] = {"method": "bm25", "terms": True, "normalize": True}

        # Check if default model should be loaded
        if not self.model and self.defaultallowed():
            self.config["path"] = "sentence-transformers/all-MiniLM-L6-v2"

            # Load dense vectors model
            self.model = self.loadvectors()

    def defaultallowed(self):
        """
        Tests if this embeddings instance can use a default model if not otherwise provided.

        Returns:
            True if a default model is allowed, False otherwise
        """

        params = [("keyword", False), ("defaults", True)]
        return all(self.config.get(key, default) == default for key, default in params)

    def loadconfig(self, path):
        """
        Loads index configuration. This method supports both config.json and config pickle files.

        Args:
            path: path to directory

        Returns:
            dict
        """

        # Configuration
        config = None

        # Determine if config is json or pickle
        jsonconfig = os.path.exists(f"{path}/config.json")

        # Set config file name
        name = "config.json" if jsonconfig else "config"

        # Load configuration
        with open(f"{path}/{name}", "r" if jsonconfig else "rb", encoding="utf-8" if jsonconfig else None) as handle:
            config = json.load(handle) if jsonconfig else pickle.load(handle)

        # Add format parameter
        config["format"] = "json" if jsonconfig else "pickle"

        # Build full path to embedding vectors file
        if config.get("storevectors"):
            config["path"] = os.path.join(path, config["path"])

        return config

    def saveconfig(self, path):
        """
        Saves index configuration. This method defaults to JSON and falls back to pickle.

        Args:
            path: path to directory

        Returns:
            dict
        """

        # Default to JSON config
        jsonconfig = self.config.get("format", "json") == "json"

        # Set config file name
        name = "config.json" if jsonconfig else "config"

        # Write configuration
        with open(f"{path}/{name}", "w" if jsonconfig else "wb", encoding="utf-8" if jsonconfig else None) as handle:
            if jsonconfig:
                # Write config as JSON
                json.dump(self.config, handle, default=str, indent=2)
            else:
                # Write config as pickle format
                pickle.dump(self.config, handle, protocol=__pickle__)

    def loadvectors(self):
        """
        Loads a vector model set in config.

        Returns:
            vector model
        """

        # Create model cache if subindexes are enabled
        if "indexes" in self.config and self.models is None:
            self.models = {}

        # Model path
        path = self.config.get("path")

        # Check if model is cached
        if self.models and path in self.models:
            return self.models[path]

        # Load and store uncached model
        model = VectorsFactory.create(self.config, self.scoring)
        if self.models is not None and path:
            self.models[path] = model

        return model

    def loadquery(self):
        """
        Loads a query model set in config.

        Returns:
            query model
        """

        if "query" in self.config:
            return Query(**self.config["query"])

        return None

    def checkarchive(self, path):
        """
        Checks if path is an archive file.

        Args:
            path: path to check

        Returns:
            (working directory, current path) if this is an archive, original path otherwise
        """

        # Create archive instance, if necessary
        self.archive = ArchiveFactory.create()

        # Check if path is an archive file
        if self.archive.isarchive(path):
            # Return temporary archive working directory and original path
            return self.archive.path(), path

        return path, None

    def createcloud(self, **cloud):
        """
        Creates a cloud instance from config.

        Args:
            cloud: cloud configuration
        """

        # Merge keyword args and keys under the cloud parameter
        config = cloud
        if "cloud" in config and config["cloud"]:
            config.update(config.pop("cloud"))

        # Create cloud instance from config and return
        return CloudFactory.create(config) if config else None

    def createann(self):
        """
        Creates an ANN from config.

        Returns:
            new ANN, if enabled in config
        """

        return ANNFactory.create(self.config) if self.config.get("path") or self.defaultallowed() else None

    def createdatabase(self):
        """
        Creates a database from config. This method will also close any existing database connection.

        Returns:
            new database, if enabled in config
        """

        # Free existing database resources
        if self.database:
            self.database.close()

        config = self.config.copy()

        # Create references to callable functions
        self.functions = Functions(self) if "functions" in config else None
        if self.functions:
            config["functions"] = self.functions(config)

        # Create database from config and return
        return DatabaseFactory.create(config)

    def creategraph(self):
        """
        Creates a graph from config.

        Returns:
            new graph, if enabled in config
        """

        if "graph" in self.config:
            # Get or create graph configuration
            config = self.config["graph"] if self.config["graph"] else {}

            # Create configuration with custom columns, if necessary
            config = self.columns(config)
            return GraphFactory.create(config)

        return None

    def createids(self, ids=None):
        """
        Creates indexids when content is disabled.

        Args:
            ids: optional ids to add

        Returns:
            new indexids, if content disabled
        """

        # Load index ids when content is disabled
        return IndexIds(self, ids) if not self.config.get("content") else None

    def createindexes(self):
        """
        Creates subindexes from config.

        Returns:
            list of subindexes
        """

        # Load subindexes
        if "indexes" in self.config:
            indexes = {}
            for index, config in self.config["indexes"].items():
                # Create index with shared model cache
                indexes[index] = Embeddings(config, models=self.models)

            # Wrap as Indexes object
            return Indexes(self, indexes)

        return None

    def createscoring(self):
        """
        Creates a scoring from config.

        Returns:
            new scoring, if enabled in config
        """

        # Free existing resources
        if self.scoring:
            self.scoring.close()

        if "scoring" in self.config:
            # Expand scoring to a dictionary, if necessary
            config = self.config["scoring"]
            config = config if isinstance(config, dict) else {"method": config}

            # Create configuration with custom columns, if necessary
            config = self.columns(config)
            return ScoringFactory.create(config)

        return None

    def columns(self, config):
        """
        Adds custom text/object column information if it's provided.

        Args:
            config: input configuration

        Returns:
            config with column information added
        """

        # Add text/object columns if custom
        if "columns" in self.config:
            # Work on copy of configuration
            config = config.copy()

            # Copy columns to config
            config["columns"] = self.config["columns"]

        return config

__init__(config=None, models=None, **kwargs)

Creates a new embeddings index. Embeddings indexes are thread-safe for read operations but writes must be synchronized.

Parameters:

Name Type Description Default
config

embeddings configuration

None
models

models cache, used for model sharing between embeddings

None
kwargs

additional configuration as keyword args

{}
Source code in txtai/embeddings/base.py
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
def __init__(self, config=None, models=None, **kwargs):
    """
    Creates a new embeddings index. Embeddings indexes are thread-safe for read operations but writes must be synchronized.

    Args:
        config: embeddings configuration
        models: models cache, used for model sharing between embeddings
        kwargs: additional configuration as keyword args
    """

    # Index configuration
    self.config = None

    # Dimensionality reduction - word vectors only
    self.reducer = None

    # Dense vector model - transforms data into similarity vectors
    self.model = None

    # Approximate nearest neighbor index
    self.ann = None

    # Index ids when content is disabled
    self.ids = None

    # Document database
    self.database = None

    # Resolvable functions
    self.functions = None

    # Graph network
    self.graph = None

    # Sparse vectors
    self.scoring = None

    # Query model
    self.query = None

    # Index archive
    self.archive = None

    # Subindexes for this embeddings instance
    self.indexes = None

    # Models cache
    self.models = models

    # Merge configuration into single dictionary
    config = {**config, **kwargs} if config and kwargs else kwargs if kwargs else config

    # Set initial configuration
    self.configure(config)

batchexplain(queries, texts=None, limit=None)

Explains the importance of each input token in text for a list of queries. This method requires either content to be enabled or texts to be provided.

Parameters:

Name Type Description Default
queries

input queries

required
texts

optional list of (text|list of tokens), otherwise runs search queries

None
limit

optional limit if texts is None

None

Returns:

Type Description

list of dict per input text per query where a higher token scores represents higher importance relative to the query

Source code in txtai/embeddings/base.py
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
def batchexplain(self, queries, texts=None, limit=None):
    """
    Explains the importance of each input token in text for a list of queries. This method requires either content to be enabled
    or texts to be provided.

    Args:
        queries: input queries
        texts: optional list of (text|list of tokens), otherwise runs search queries
        limit: optional limit if texts is None

    Returns:
        list of dict per input text per query where a higher token scores represents higher importance relative to the query
    """

    return Explain(self)(queries, texts, limit)

batchsearch(queries, limit=None, weights=None, index=None, parameters=None, graph=False)

Finds documents most similar to the input queries. This method will run either an index search or an index + database search depending on if a database is available.

Parameters:

Name Type Description Default
queries

input queries

required
limit

maximum results

None
weights

hybrid score weights, if applicable

None
index

index name, if applicable

None
parameters

list of dicts of named parameters to bind to placeholders

None
graph

return graph results if True

False

Returns:

Type Description

list of (id, score) per query for index search

list of dict per query for an index + database search

list of graph per query when graph is set to True

Source code in txtai/embeddings/base.py
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
def batchsearch(self, queries, limit=None, weights=None, index=None, parameters=None, graph=False):
    """
    Finds documents most similar to the input queries. This method will run either an index search
    or an index + database search depending on if a database is available.

    Args:
        queries: input queries
        limit: maximum results
        weights: hybrid score weights, if applicable
        index: index name, if applicable
        parameters: list of dicts of named parameters to bind to placeholders
        graph: return graph results if True

    Returns:
        list of (id, score) per query for index search
        list of dict per query for an index + database search
        list of graph per query when graph is set to True
    """

    # Determine if graphs should be returned
    graph = graph if self.graph else False

    # Execute search
    results = Search(self, graph)(queries, limit, weights, index, parameters)

    # Create subgraphs using results, if necessary
    return [self.graph.filter(x) for x in results] if graph else results

batchsimilarity(queries, data)

Computes the similarity between list of queries and list of data. Returns a list of (id, score) sorted by highest score per query, where id is the index in data.

Parameters:

Name Type Description Default
queries

input queries

required
data

list of data

required

Returns:

Type Description

list of (id, score) per query

Source code in txtai/embeddings/base.py
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
def batchsimilarity(self, queries, data):
    """
    Computes the similarity between list of queries and list of data. Returns a list
    of (id, score) sorted by highest score per query, where id is the index in data.

    Args:
        queries: input queries
        data: list of data

    Returns:
        list of (id, score) per query
    """

    # Convert queries to embedding vectors
    queries = self.batchtransform(((None, query, None) for query in queries), "query")
    data = self.batchtransform(((None, row, None) for row in data), "data")

    # Dot product on normalized vectors is equal to cosine similarity
    scores = np.dot(queries, data.T).tolist()

    # Add index and sort desc based on score
    return [sorted(enumerate(score), key=lambda x: x[1], reverse=True) for score in scores]

batchterms(queries)

Extracts keyword terms from a list of queries.

Parameters:

Name Type Description Default
queries

list of queries

required

Returns:

Type Description

list of queries reduced down to keyword term strings

Source code in txtai/embeddings/base.py
484
485
486
487
488
489
490
491
492
493
494
495
def batchterms(self, queries):
    """
    Extracts keyword terms from a list of queries.

    Args:
        queries: list of queries

    Returns:
        list of queries reduced down to keyword term strings
    """

    return Terms(self)(queries)

batchtransform(documents, category=None)

Transforms documents into embeddings vectors.

Parameters:

Name Type Description Default
documents

iterable of (id, data, tags), (id, data) or data

required
category

category for instruction-based embeddings

None

Returns:

Type Description

embeddings vectors

Source code in txtai/embeddings/base.py
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
def batchtransform(self, documents, category=None):
    """
    Transforms documents into embeddings vectors.

    Args:
        documents: iterable of (id, data, tags), (id, data) or data
        category: category for instruction-based embeddings

    Returns:
        embeddings vectors
    """

    # Initialize default parameters, if necessary
    self.defaults()

    # Default vector model, if necessary
    model = self.model if self.model else next(iter(self.models.values()))

    # Convert documents into sentence embeddings
    embeddings = model.batchtransform(Stream(self)(documents), category)

    # Reduce the dimensionality of the embeddings. Scale the embeddings using this
    # model to reduce the noise of common but less relevant terms.
    if self.reducer:
        self.reducer(embeddings)

    return embeddings

close()

Closes this embeddings index and frees all resources.

Source code in txtai/embeddings/base.py
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
def close(self):
    """
    Closes this embeddings index and frees all resources.
    """

    self.ann, self.config, self.graph, self.archive = None, None, None, None
    self.reducer, self.query, self.model, self.models = None, None, None, None
    self.ids = None

    # Close database connection if open
    if self.database:
        self.database.close()
        self.database, self.functions = None, None

    # Close scoring instance if open
    if self.scoring:
        self.scoring.close()
        self.scoring = None

    # Close indexes if open
    if self.indexes:
        self.indexes.close()
        self.indexes = None

count()

Total number of elements in this embeddings index.

Returns:

Type Description

number of elements in this embeddings index

Source code in txtai/embeddings/base.py
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
def count(self):
    """
    Total number of elements in this embeddings index.

    Returns:
        number of elements in this embeddings index
    """

    if self.ann:
        return self.ann.count()
    if self.scoring:
        return self.scoring.count()
    if self.database:
        return self.database.count()
    if self.ids:
        return len([uid for uid in self.ids if uid is not None])

    # Default to 0 when no suitable method found
    return 0

delete(ids)

Deletes from an embeddings index. Returns list of ids deleted.

Parameters:

Name Type Description Default
ids

list of ids to delete

required

Returns:

Type Description

list of ids deleted

Source code in txtai/embeddings/base.py
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
def delete(self, ids):
    """
    Deletes from an embeddings index. Returns list of ids deleted.

    Args:
        ids: list of ids to delete

    Returns:
        list of ids deleted
    """

    # List of internal indices for each candidate id to delete
    indices = []

    # List of deleted ids
    deletes = []

    if self.database:
        # Retrieve indexid-id mappings from database
        ids = self.database.ids(ids)

        # Parse out indices and ids to delete
        indices = [i for i, _ in ids]
        deletes = sorted(set(uid for _, uid in ids))

        # Delete ids from database
        self.database.delete(deletes)
    elif self.ann or self.scoring:
        # Find existing ids
        for uid in ids:
            indices.extend([index for index, value in enumerate(self.ids) if uid == value])

        # Clear embeddings ids
        for index in indices:
            deletes.append(self.ids[index])
            self.ids[index] = None

    # Delete indices for all indexes and data stores
    if indices:
        # Delete ids from ann
        if self.isdense():
            self.ann.delete(indices)

        # Delete ids from scoring
        if self.issparse():
            self.scoring.delete(indices)

        # Delete ids from subindexes
        if self.indexes:
            self.indexes.delete(indices)

        # Delete ids from graph
        if self.graph:
            self.graph.delete(indices)

    return deletes

exists(path=None, cloud=None, **kwargs)

Checks if an index exists at path.

Parameters:

Name Type Description Default
path

input path

None
cloud

cloud storage configuration

None
kwargs

additional configuration as keyword args

{}

Returns:

Type Description

True if index exists, False otherwise

Source code in txtai/embeddings/base.py
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
def exists(self, path=None, cloud=None, **kwargs):
    """
    Checks if an index exists at path.

    Args:
        path: input path
        cloud: cloud storage configuration
        kwargs: additional configuration as keyword args

    Returns:
        True if index exists, False otherwise
    """

    # Check if this exists in a cloud instance
    cloud = self.createcloud(cloud=cloud, **kwargs)
    if cloud:
        return cloud.exists(path)

    # Check if this is an archive file and exists
    path, apath = self.checkarchive(path)
    if apath:
        return os.path.exists(apath)

    # Return true if path has a config.json or config file with an offset set
    return path and (os.path.exists(f"{path}/config.json") or os.path.exists(f"{path}/config")) and "offset" in self.loadconfig(path)

explain(query, texts=None, limit=None)

Explains the importance of each input token in text for a query. This method requires either content to be enabled or texts to be provided.

Parameters:

Name Type Description Default
query

input query

required
texts

optional list of (text|list of tokens), otherwise runs search query

None
limit

optional limit if texts is None

None

Returns:

Type Description

list of dict per input text where a higher token scores represents higher importance relative to the query

Source code in txtai/embeddings/base.py
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
def explain(self, query, texts=None, limit=None):
    """
    Explains the importance of each input token in text for a query. This method requires either content to be enabled
    or texts to be provided.

    Args:
        query: input query
        texts: optional list of (text|list of tokens), otherwise runs search query
        limit: optional limit if texts is None

    Returns:
        list of dict per input text where a higher token scores represents higher importance relative to the query
    """

    results = self.batchexplain([query], texts, limit)
    return results[0] if results else results

index(documents, reindex=False)

Builds an embeddings index. This method overwrites an existing index.

Parameters:

Name Type Description Default
documents

iterable of (id, data, tags), (id, data) or data

required
reindex

if this is a reindex operation in which case database creation is skipped, defaults to False

False
Source code in txtai/embeddings/base.py
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
def index(self, documents, reindex=False):
    """
    Builds an embeddings index. This method overwrites an existing index.

    Args:
        documents: iterable of (id, data, tags), (id, data) or data
        reindex: if this is a reindex operation in which case database creation is skipped, defaults to False
    """

    # Initialize index
    self.initindex(reindex)

    # Create transform and stream
    transform = Transform(self, Action.REINDEX if reindex else Action.INDEX)
    stream = Stream(self, Action.REINDEX if reindex else Action.INDEX)

    with tempfile.NamedTemporaryFile(mode="wb", suffix=".npy") as buffer:
        # Load documents into database and transform to vectors
        ids, dimensions, embeddings = transform(stream(documents), buffer)
        if embeddings is not None:
            # Build LSA model (if enabled). Remove principal components from embeddings.
            if self.config.get("pca"):
                self.reducer = Reducer(embeddings, self.config["pca"])
                self.reducer(embeddings)

            # Save index dimensions
            self.config["dimensions"] = dimensions

            # Create approximate nearest neighbor index
            self.ann = self.createann()

            # Add embeddings to the index
            self.ann.index(embeddings)

        # Save indexids-ids mapping for indexes with no database, except when this is a reindex
        if ids and not reindex and not self.database:
            self.ids = self.createids(ids)

    # Index scoring, if necessary
    # This must occur before graph index in order to be available to the graph
    if self.issparse():
        self.scoring.index()

    # Index subindexes, if necessary
    if self.indexes:
        self.indexes.index()

    # Index graph, if necessary
    if self.graph:
        self.graph.index(Search(self, True), Ids(self), self.batchsimilarity)

info()

Prints the current embeddings index configuration.

Source code in txtai/embeddings/base.py
679
680
681
682
683
684
685
686
def info(self):
    """
    Prints the current embeddings index configuration.
    """

    if self.config:
        # Print configuration
        print(json.dumps(self.config, sort_keys=True, default=str, indent=2))

isdense()

Checks if this instance has an associated ANN instance.

Returns:

Type Description

True if this instance has an associated ANN, False otherwise

Source code in txtai/embeddings/base.py
698
699
700
701
702
703
704
705
706
def isdense(self):
    """
    Checks if this instance has an associated ANN instance.

    Returns:
        True if this instance has an associated ANN, False otherwise
    """

    return self.ann is not None

issparse()

Checks if this instance has an associated scoring instance with term indexing enabled.

Returns:

Type Description

True if term index is enabled, False otherwise

Source code in txtai/embeddings/base.py
688
689
690
691
692
693
694
695
696
def issparse(self):
    """
    Checks if this instance has an associated scoring instance with term indexing enabled.

    Returns:
        True if term index is enabled, False otherwise
    """

    return self.scoring and self.scoring.hasterms()

isweighted()

Checks if this instance has an associated scoring instance with term weighting enabled.

Returns:

Type Description

True if term weighting is enabled, False otherwise

Source code in txtai/embeddings/base.py
708
709
710
711
712
713
714
715
716
def isweighted(self):
    """
    Checks if this instance has an associated scoring instance with term weighting enabled.

    Returns:
        True if term weighting is enabled, False otherwise
    """

    return self.scoring and not self.scoring.hasterms()

load(path=None, cloud=None, config=None, **kwargs)

Loads an existing index from path.

Parameters:

Name Type Description Default
path

input path

None
cloud

cloud storage configuration

None
config

configuration overrides

None
kwargs

additional configuration as keyword args

{}
Source code in txtai/embeddings/base.py
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
def load(self, path=None, cloud=None, config=None, **kwargs):
    """
    Loads an existing index from path.

    Args:
        path: input path
        cloud: cloud storage configuration
        config: configuration overrides
        kwargs: additional configuration as keyword args
    """

    # Load from cloud, if configured
    cloud = self.createcloud(cloud=cloud, **kwargs)
    if cloud:
        path = cloud.load(path)

    # Check if this is an archive file and extract
    path, apath = self.checkarchive(path)
    if apath:
        self.archive.load(apath)

    # Load index configuration
    self.config = self.loadconfig(path)

    # Apply config overrides
    self.config = {**self.config, **config} if config else self.config

    # Approximate nearest neighbor index - stores dense vectors
    self.ann = self.createann()
    if self.ann:
        self.ann.load(f"{path}/embeddings")

    # Dimensionality reduction model - word vectors only
    if self.config.get("pca"):
        self.reducer = Reducer()
        self.reducer.load(f"{path}/lsa")

    # Index ids when content is disabled
    self.ids = self.createids()
    if self.ids:
        self.ids.load(f"{path}/ids")

    # Document database - stores document content
    self.database = self.createdatabase()
    if self.database:
        self.database.load(f"{path}/documents")

    # Sparse vectors - stores term sparse arrays
    self.scoring = self.createscoring()
    if self.scoring:
        self.scoring.load(f"{path}/scoring")

    # Subindexes
    self.indexes = self.createindexes()
    if self.indexes:
        self.indexes.load(f"{path}/indexes")

    # Graph network - stores relationships
    self.graph = self.creategraph()
    if self.graph:
        self.graph.load(f"{path}/graph")

    # Dense vectors - transforms data to embeddings vectors
    self.model = self.loadvectors()

    # Query model
    self.query = self.loadquery()

reindex(config=None, function=None, **kwargs)

Recreates embeddings index using config. This method only works if document content storage is enabled.

Parameters:

Name Type Description Default
config

new config

None
function

optional function to prepare content for indexing

None
kwargs

additional configuration as keyword args

{}
Source code in txtai/embeddings/base.py
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
def reindex(self, config=None, function=None, **kwargs):
    """
    Recreates embeddings index using config. This method only works if document content storage is enabled.

    Args:
        config: new config
        function: optional function to prepare content for indexing
        kwargs: additional configuration as keyword args
    """

    if self.database:
        # Merge configuration into single dictionary
        config = {**config, **kwargs} if config and kwargs else config if config else kwargs

        # Keep content and objects parameters to ensure database is preserved
        config["content"] = self.config["content"]
        if "objects" in self.config:
            config["objects"] = self.config["objects"]

        # Reset configuration
        self.configure(config)

        # Reset function references
        if self.functions:
            self.functions.reset()

        # Reindex
        if function:
            self.index(function(self.database.reindex(self.config)), True)
        else:
            self.index(self.database.reindex(self.config), True)

save(path, cloud=None, **kwargs)

Saves an index in a directory at path unless path ends with tar.gz, tar.bz2, tar.xz or zip. In those cases, the index is stored as a compressed file.

Parameters:

Name Type Description Default
path

output path

required
cloud

cloud storage configuration

None
kwargs

additional configuration as keyword args

{}
Source code in txtai/embeddings/base.py
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
def save(self, path, cloud=None, **kwargs):
    """
    Saves an index in a directory at path unless path ends with tar.gz, tar.bz2, tar.xz or zip.
    In those cases, the index is stored as a compressed file.

    Args:
        path: output path
        cloud: cloud storage configuration
        kwargs: additional configuration as keyword args
    """

    if self.config:
        # Check if this is an archive file
        path, apath = self.checkarchive(path)

        # Create output directory, if necessary
        os.makedirs(path, exist_ok=True)

        # Copy sentence vectors model
        if self.config.get("storevectors"):
            shutil.copyfile(self.config["path"], os.path.join(path, os.path.basename(self.config["path"])))

            self.config["path"] = os.path.basename(self.config["path"])

        # Save index configuration
        self.saveconfig(path)

        # Save approximate nearest neighbor index
        if self.ann:
            self.ann.save(f"{path}/embeddings")

        # Save dimensionality reduction model (word vectors only)
        if self.reducer:
            self.reducer.save(f"{path}/lsa")

        # Save index ids
        if self.ids:
            self.ids.save(f"{path}/ids")

        # Save document database
        if self.database:
            self.database.save(f"{path}/documents")

        # Save scoring index
        if self.scoring:
            self.scoring.save(f"{path}/scoring")

        # Save subindexes
        if self.indexes:
            self.indexes.save(f"{path}/indexes")

        # Save graph
        if self.graph:
            self.graph.save(f"{path}/graph")

        # If this is an archive, save it
        if apath:
            self.archive.save(apath)

        # Save to cloud, if configured
        cloud = self.createcloud(cloud=cloud, **kwargs)
        if cloud:
            cloud.save(apath if apath else path)

score(documents)

Builds a term weighting scoring index. Only used by word vectors models.

Parameters:

Name Type Description Default
documents

iterable of (id, data, tags), (id, data) or data

required
Source code in txtai/embeddings/base.py
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def score(self, documents):
    """
    Builds a term weighting scoring index. Only used by word vectors models.

    Args:
        documents: iterable of (id, data, tags), (id, data) or data
    """

    # Build scoring index for word vectors term weighting
    if self.isweighted():
        self.scoring.index(Stream(self)(documents))

search(query, limit=None, weights=None, index=None, parameters=None, graph=False)

Finds documents most similar to the input query. This method will run either an index search or an index + database search depending on if a database is available.

Parameters:

Name Type Description Default
query

input query

required
limit

maximum results

None
weights

hybrid score weights, if applicable

None
index

index name, if applicable

None
parameters

dict of named parameters to bind to placeholders

None
graph

return graph results if True

False

Returns:

Type Description

list of (id, score) for index search

list of dict for an index + database search

graph when graph is set to True

Source code in txtai/embeddings/base.py
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
def search(self, query, limit=None, weights=None, index=None, parameters=None, graph=False):
    """
    Finds documents most similar to the input query. This method will run either an index search
    or an index + database search depending on if a database is available.

    Args:
        query: input query
        limit: maximum results
        weights: hybrid score weights, if applicable
        index: index name, if applicable
        parameters: dict of named parameters to bind to placeholders
        graph: return graph results if True

    Returns:
        list of (id, score) for index search
        list of dict for an index + database search
        graph when graph is set to True
    """

    results = self.batchsearch([query], limit, weights, index, [parameters], graph)
    return results[0] if results else results

similarity(query, data)

Computes the similarity between query and list of data. Returns a list of (id, score) sorted by highest score, where id is the index in data.

Parameters:

Name Type Description Default
query

input query

required
data

list of data

required

Returns:

Type Description

list of (id, score)

Source code in txtai/embeddings/base.py
400
401
402
403
404
405
406
407
408
409
410
411
412
413
def similarity(self, query, data):
    """
    Computes the similarity between query and list of data. Returns a list of
    (id, score) sorted by highest score, where id is the index in data.

    Args:
        query: input query
        data: list of data

    Returns:
        list of (id, score)
    """

    return self.batchsimilarity([query], data)[0]

terms(query)

Extracts keyword terms from a query.

Parameters:

Name Type Description Default
query

input query

required

Returns:

Type Description

query reduced down to keyword terms

Source code in txtai/embeddings/base.py
471
472
473
474
475
476
477
478
479
480
481
482
def terms(self, query):
    """
    Extracts keyword terms from a query.

    Args:
        query: input query

    Returns:
        query reduced down to keyword terms
    """

    return self.batchterms([query])[0]

transform(document)

Transforms document into an embeddings vector.

Parameters:

Name Type Description Default
documents

iterable of (id, data, tags), (id, data) or data

required

Returns:

Type Description

embeddings vector

Source code in txtai/embeddings/base.py
289
290
291
292
293
294
295
296
297
298
299
300
def transform(self, document):
    """
    Transforms document into an embeddings vector.

    Args:
        documents: iterable of (id, data, tags), (id, data) or data

    Returns:
        embeddings vector
    """

    return self.batchtransform([document])[0]

upsert(documents)

Runs an embeddings upsert operation. If the index exists, new data is appended to the index, existing data is updated. If the index doesn't exist, this method runs a standard index operation.

Parameters:

Name Type Description Default
documents

iterable of (id, data, tags), (id, data) or data

required
Source code in txtai/embeddings/base.py
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
def upsert(self, documents):
    """
    Runs an embeddings upsert operation. If the index exists, new data is
    appended to the index, existing data is updated. If the index doesn't exist,
    this method runs a standard index operation.

    Args:
        documents: iterable of (id, data, tags), (id, data) or data
    """

    # Run standard insert if index doesn't exist or it has no records
    if not self.count():
        self.index(documents)
        return

    # Create transform and stream
    transform = Transform(self, Action.UPSERT)
    stream = Stream(self, Action.UPSERT)

    with tempfile.NamedTemporaryFile(mode="wb", suffix=".npy") as buffer:
        # Load documents into database and transform to vectors
        ids, _, embeddings = transform(stream(documents), buffer)
        if embeddings is not None:
            # Remove principal components from embeddings, if necessary
            if self.reducer:
                self.reducer(embeddings)

            # Append embeddings to the index
            self.ann.append(embeddings)

        # Save indexids-ids mapping for indexes with no database
        if ids and not self.database:
            self.ids = self.createids(self.ids + ids)

    # Scoring upsert, if necessary
    # This must occur before graph upsert in order to be available to the graph
    if self.issparse():
        self.scoring.upsert()

    # Subindexes upsert, if necessary
    if self.indexes:
        self.indexes.upsert()

    # Graph upsert, if necessary
    if self.graph:
        self.graph.upsert(Search(self, True), Ids(self), self.batchsimilarity)