Why an old index cannot serve a new model
An embedding model maps text to a point in a space it defines itself. The axes of that space have no shared meaning with the axes of another model. Two consequences follow. First, the vector length usually differs: a 768-dimension index cannot store a 1536-dimension vector, and the database rejects the write. Second, even when two models happen to emit the same number of dimensions, a query vector from model B lands in a region that model A never used. Cosine similarity still returns a number, and that number is meaningless. This is the dangerous case, because nothing errors: recall simply drops and nobody notices until users complain.
What actually breaks
- Index dimension. The collection has a fixed vector size. A new model with a different width needs a new collection.
- Similarity thresholds. Any hard-coded cutoff, such as "keep chunks above 0.75", was tuned on the old model's score distribution. Re-tune it on the new one.
- Mixed collections. Writing new documents with the new model into the old collection produces a silently corrupt index. Version the collection name instead.
- Query-time consistency. The query must be embedded with the same model that embedded the documents. A deployment that updates one side before the other fails.
Two things do not need to change. A reranker scores text pairs, not vectors, so it survives the migration. A keyword index such as BM25 over the original text also survives, which is why keeping the raw text next to the vector matters.
Plan the migration as a rebuild
- Keep the source text and metadata. Without the original chunks you cannot re-embed; you can only re-crawl. Check this first.
- Freeze the chunking. Re-embed the same chunk boundaries so that retrieval results stay comparable. Change one variable at a time.
- Create a versioned collection such as
docs_v2_<model>. Write the model name and its version into the collection metadata. - Re-embed in batches and record the count of chunks written against the count of chunks read. The two numbers must match at the end.
- Evaluate both indexes on the same query set before you move traffic. Compare recall at k, not average similarity scores.
- Cut over by alias. Point the application at the new collection, keep the old one for a rollback window, then delete it.
Estimate the cost before you start
The bill is the number of chunks multiplied by the average tokens per chunk. A corpus of 200,000 chunks at 400 tokens each is 80,000,000 tokens, and embedding prices are quoted per million tokens. Multiply that figure by the current price of both the old and the new model if you plan to keep both indexes warm. Add storage for the second index, because both copies exist during the migration.
Time is the other budget. Re-embedding is rate limited by the provider, so divide the token total by your allowed tokens per minute to get a floor on the wall-clock time, then add the indexing time of the vector database.
Verify before cutover
- Query 20 to 50 real questions that you already know the correct source chunk for.
- Check that the correct chunk appears in the top k for the new index at least as often as for the old one.
- Re-tune any similarity cutoff on the new score distribution.
- Compare end-to-end answers, not only retrieval, because a different model changes what the generator sees.
If the new model wins on recall but loses on answer quality, the problem is usually in the prompt or the context ordering, not in the embedding. Evaluate the two stages separately before you blame the vectors.