projects / vecsearch.git / summary
| description | CLI to query and update a search engine based on a BERT transformer model and a vector store |
| last change | Sat, 30 Nov 2024 15:01:35 +0000 (09:01 -0600) |
| URL | https://git.humopery.space/vecsearch.git |
A tool to manage a basic vector-based search index with candle and pgvector. It can create the index, add documents and run searches.
Borrowed heavily from:
The postgresql database must have the pgvector extension installed. The user must have superuser in the target database.
If the target database doesn’t exist yet, then the user must have createdb permission.
$ ./vecsearch init-database --help
Initialize the database when the database or table doesn't exist already
Usage: vecsearch init-database [OPTIONS] --dbpassword <DBPASSWORD>
Options:
--dbname <DBNAME> [default: vsearch]
--dbhost <DBHOST> [default: localhost]
--dbuser <DBUSER> [default: vsmigrator]
--dbpassword <DBPASSWORD> [env: DBPASSWORD=]
-h, --help Print help
For example:
$ export DBPASSWORD=$(gpg -d pw-vsmigrator.gpg)
$ vecsearch init-database
maybe creating the database
database vsearch exists already
maybe creating database objects
Download the model files. This command is optional since the model files can be downloaded lazily by the index and search actions.
$ ./vecsearch init-model
A document is a regular file.
The user for this operation requires only write access to the table (not superuser).
Specifying multiple files is more efficient than indexing one file in each invocation.
$ export DBPASSWORD=$(gpg -d pw-vsmigrator.gpg)
$ vecsearch index --file testdata/0 --file testdata/1
indexing file(s)
Loaded and encoded 58.565µs
Took 15.628167ms
Loaded and encoded 55.513µs
Took 8.018493ms
Return the top five matches for the given search. Note the search is semantic so the right document is returned for e.g. “meow” or “canine” even without the documents containing those words.
The user for this operation needs only read access (not write or superuser).
$ export DBPASSWORD=$(gpg -d pw-vsmigrator.gpg)
$ vecsearch search --search feline
searching for document matches
Loaded and encoded 49.306µs
Took 14.452557ms
The cat sits outside
The cat plays in the garden
I love pasta
Do you like pizza?
The new movie is so great
| 13 months ago | main | shortlog | log | tree |