I still don't like dataframes but oh my God Polars is so much better than pandas.

I was doing some time series calculations, simple equity price adjustments basically, in Polars and my two thoughts were:

- WTF, I can actually read the code and test it.

- it's running so fast it seems like it's broken.

Really cool article, I've enjoyed your work for a long time. You might add a note for those jumping into a sqlite implementation, that duckdb reads parquet and launched a few vector similarity functions which cover this use-case perfectly:

https://duckdb.org/2024/05/03/vector-similarity-search-vss.h...

The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times.

If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays.

Personally I prefer using SQLite with usearch extension. Binary vectors then rerank top 100 with float32. It’s about 2 ms for ~20k items, which beats LanceDB in my tests. Maybe Lance wins on bigger collections. But for my use case it works great, as each user has their own dedicated SQLite file.

For portability there’s Litestream.

Check out Unum’s usearch. It beats anything, and is super easy to use. It just does exactly what you need.

https://github.com/unum-cloud/usearch

In 2017 I was working on a model trainer for text classification and sequence labeling [1] that had limited success because the models weren't good enough.

I have a minilm + pooling + svm classifier which works pretty well for some things (topics, "will I like this article?") but doesn't work so well for sentiment, emotional tone and other things where the order of the words matter. I'm planning to upgrade my current classifier's front end to use ModernBert and add an LSTM-based back end that I think will equal or beat fine-tuned BERT and, more importantly, can be trained reliably with early stopping. I'd like to open source the thing, focused on reliability, because I'm an application programmer at heart.

I want it to provide an interface which is text-in and labels-out and hide the embeddings from most users but I'm definitely thinking about how to handle them, and there's the worse problem here that the LSTM needs a vector for each token, not each document, so text gets puffed up by a factor of 1000 or so which is not insurmountable (1 MB of training text puffs up to 1 GB of vectors)

Since it's expensive to compute the embeddings and expensive to store them I'm thinking about whether and how to cache them, considering that I expect to present the same samples to the trainer multiple times and to do a lot of model selection in the process of model development (e.g. what exact shape LSTM to to use) and in the case of end-user training (it will probably try a few models, not least do a shootout between the expensive model and a cheap model)_

[1] think of a "magic magic marker" which learns to mark up text the same way you do; this could mark "needless words" you could delete from a title, parts of speech, named entities, etc.

Lots of great findings

---

I'm curious if anyone knows whether it is better to pass structured data or unstructured data to embedding api's? If I ask ChatGPT, it says it is better to send unstructured data. (looking at the authors github, it looks like he generated embeddings from json strings)

My use case is for jsonresume, I am creating embeddings by sending full json versions as strings, but I've been experimenting with using models to translate resume.json's into full text versions first before creating embeddings. The results seem to be better but I haven't seen any concrete opinions on this.

My understanding is that unstructured data is better because it contains textual/semantic meaning because of natural lanaguage aka

  skills: ['Javascript', 'Python']

  Thomas excels at Javascript and Python

I'm a huge fan of polars, but I hadn't considered using it to store embeddings in this way (I've been fiddling with sqlite-vec). Seems like an interesting idea indeed.

>The second incorrect method to save a matrix of embeddings to disk is to save it as a Python pickle object [...] But it comes with two major caveats: pickled files are a massive security risk as they can execute arbitrary code, and the pickled file may not be guaranteed to be able to be opened on other machines or Python versions. It’s 2025, just stop pickling if you can.

Security: absolutely.

Portability: who cares? Frameworks move so quickly that unless you carry your whole dependency graph between machines you will not get bit compatible results with even minor version changes. It's a dirty secret that no one seems to want to fix or care about.

In short: everything is so fucked that pickle + conda is more than good enough for whatever project you want to serve to >10,000 users.

The trouble with Parquet (and columnar storage) in ML is,

1. You don't really care too-much about accessing subsets of columns

2. You can't easily append stuff to closed Parquet files.

3. Batched-row access is presumably slower due to lower cache-hits.

It's okay for map-reduce style stuff where this doesn't matter, but in ML these limitations are an annoyance.

HDF5 (or Zarr, less portably) solves some/many of these issues but it's not quite a settled affair.

Nice read. I agree that for a lot of hobby use cases you can just load the embeddings from parquet and compute the similarities in-memory.

To find similarity between my blogposts [1] I wanted to experiment with a local vector database and found ChromaDB fairly easy to use (similar to SQLite just a file on your machine).

[1] https://staticnotes.org/posts/how-recommendations-work/

If you want to try it out. Can lazily load from HF and apply filtering this way.

  df = (
    pl.scan_parquet('hf://datasets/minimaxir/mtg-embeddings/mtg_embeddings.parquet')
    .filter(
        pl.col("type").str.contains("Sorcery"),
        pl.col("manaCost").str.contains("B"),
    )
    .collect()

Polars is awesome to use, would highly recommend. Single node it is excellent at saturating CPUs, if you need to distribute the work put it in a Ray Actor with some POLARS_MAX_THREADS applied depending on how much it saturates a single node.

Polars + Parquet is awesome for portability and performance. This post focused on python portability, but Polars has an easy-to-use Rust API for embedding the engine all over the place.

For another library that has great performance and features like full text indexing and the ability to version changes I’d recommend lancedb https://lancedb.github.io/lancedb/

Yes, it’s a vector database and has more complexity. But you can use it without creating indexes and it has excellent polars and pandas zero copy arrow support also.

At 33k items in memory is quite fast, 10 ms is very responsive. With 10x/330k items given same hardware the expected time is 1 second. That might be too slow for some applications (but not all). Especially if one just does retrieval of a rather small amount of matches, an index will help a lot for 100k++ datasets.

To the second footnote: you could utilize Polar's lazyframe API to do that cosine similarity in a streaming fashion for large files.

Wow! How much did this cost you in GPU credits? And did you consider using your MacBook?

This is pretty neat.

IMO a hindrance to this was lack of built-in fixed-size list array support in the Arrow format, until recently. Some implementations/clients supported it, while others didn't. Else, it could have been used as the default storage format for numpy arrays, torch tensors, too.

(You could always store arrays as variable length list arrays with fixed strides and handle the conversion).

Is your example of a float32 number correct, holding 24 ascii char representation? I had thought single-precision gonna be 7 digits and the exponent, sign and exp sign. Something like 7+2+1+1 or 10 char ascii representation? Rather than the 24 you mentioned?

Since we are talking about an embedded solution shouldn't the benchmark be something like sqlite with a vector extension or lancedb?

or you could just use postgres + pgvector? which many apps already have installed by default.

Parquet is fine and all, but I love the simplicity and simple interoperability of CSV.

You can save a huge amount of overhead just by base64 encoding the vectors, they aren't exactly human readable anyway.

I imagine the resulting file would only be approximately 33% larger than the pickle version.