from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",   # or "text-embedding-3-large"
    # dimensions=1024,                # optional reduction (Matryoshka)
    openai_api_key="..."
)

vector = embeddings.embed_query("What is LangGraph?")
print(len(vector))  # 1536

Local Embedding Models

from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-large-en-v1.5",   # strong performer
    # model_name="intfloat/e5-mistral-7b-instruct",
    # model_name="Qwen/Qwen3-Embedding-0.6B",
    model_kwargs={"device": "cuda" if torch.cuda.is_available() else "cpu"},
    encode_kwargs={"normalize_embeddings": True}   # important for cosine similarity
)

# Batch embedding
texts = ["Document one...", "Document two..."]
vectors = embeddings.embed_documents(texts)

Practical Embedding: Model Selection and Chunking Strategy

Local vs. API Embedding Models: Which to Choose

The honest answer is: it depends on where you are in the project lifecycle and what your constraints are. Neither approach is universally better.

Models worth running locally

Not all open-source embedding models are worth the operational overhead of self-hosting. These are the ones that justify it:

BGE-M3 and Qwen3-Embedding are the most compelling local options right now. BGE-M3 supports 100+ languages and an 8192 token window, making it competitive with paid APIs on most benchmarks. Qwen3-Embedding-0.6B is remarkably capable for its size — 1.2GB, 32k token window, multilingual — and runs acceptably on CPU for moderate workloads.

Nomic Embed is worth knowing about specifically because it supports MRL (you can truncate to 256 or 512 dims), is only 270MB, and has a generous 8192 token window. It is an excellent choice when you need something lightweight that punches above its weight class.

When local models make sense

• Data privacy is non-negotiable. Your documents never leave your machine or infrastructure. For legal, medical, financial, or proprietary codebases, this is often a hard requirement — not a preference.
• High volume at low cost. If you are embedding millions of chunks, API costs compound quickly. At OpenAI's pricing, 10 million chunks of ~500 tokens each costs roughly $20–50 depending on the model. Self-hosted is effectively free after infrastructure.
• Offline or air-gapped environments. Government, defence, and regulated industries often prohibit external API calls entirely. Local models are the only option.
• Latency control. You are not subject to API rate limits, network latency, or third-party downtime. Useful in real-time applications where p99 latency matters.
• Experimentation and development. No API key, no quota, no cost per call. Iterate freely.

When paid API models make sense

• You are in early development or prototyping. Two lines of code, no infrastructure, no model management. OpenAI embeddings are the fastest path from zero to a working RAG pipeline.
• Quality is the top priority and volume is moderate. text-embedding-3-large (3072 dims) still leads most benchmarks for English retrieval quality. If your corpus is a few hundred thousand chunks, the API cost is modest and the quality advantage may be meaningful.
• Your team has no ML infrastructure experience. Running local models well in production requires model serving (Triton, TorchServe, or a simple FastAPI wrapper), hardware provisioning, monitoring, and update management. This is real operational work. If your team's strength is product engineering rather than MLOps, the API eliminates an entire layer of complexity.
• You need guaranteed uptime SLAs. OpenAI and Voyage AI offer reliability guarantees and global infrastructure. A self-hosted model is only as reliable as the machine it runs on.

The hybrid approach (recommended for most teams)

Use local models during development and switch to API models in production — or the reverse. Because LangChain abstracts the embedding interface, swapping models requires changing one line:

# development — free, local, no quota
embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-small-en-v1.5",
    model_kwargs={"device": "cpu"},
    encode_kwargs={"normalize_embeddings": True}
)

# production — switch to API with one line change
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

The critical rule: use the same model for both indexing and querying within any given environment. Never index with one model and query with another — the vectors are incompatible.

A common real-world pattern is:

• Development: bge-small-en-v1.5 locally — fast, free, good enough to test the pipeline
• Staging: bge-large-en-v1.5 or BGE-M3 in Docker — production-equivalent quality, no API cost
• Production: evaluate both text-embedding-3-small and BGE-M3 on your actual data, then pick based on quality benchmarks and cost projections at your expected volume

Do you have strict data privacy requirements?
  └── Yes → local model, no exception

Are you prototyping or in early development?
  └── Yes → OpenAI API, get moving fast

Will you embed > 5 million chunks per month?
  └── Yes → local model, the cost math favours it

Do you need multilingual or 8k+ token windows on a budget?
  └── Yes → BGE-M3 or Qwen3-Embedding locally

Is your team small with no MLOps experience?
  └── Yes → start with API, revisit when volume grows

The bottom line: for learning and building your first production RAG system, start with text-embedding-3-small via the OpenAI API — it removes all infrastructure friction. Once your pipeline is proven and your volume or privacy requirements justify it, migrate to BGE-M3 or Qwen3-Embedding self-hosted. The LangChain abstraction makes that migration straightforward.

Token window size and why it matters

Every embedding model has a maximum input length , its token window , and this is one of the first things to check before you decide on your chunk size. If a chunk exceeds the model's window, the text is silently truncated at that boundary. The embedding is computed only on the portion that fits, and the rest is simply discarded without any warning. This is a silent failure: your pipeline runs, no error is thrown, but chunks longer than the window are partially unrepresented.

Token window sizes vary significantly across models:

Notice that BGE-large-en-v1.5 , a popular and high-quality local model , has a window of only 512 tokens. If you chunk at 800 tokens with this model, every chunk over 512 tokens gets silently truncated. This is a common and costly mistake.

You can retrieve the token window programmatically rather than relying on documentation:

from transformers import AutoTokenizer

model_name = "BAAI/bge-large-en-v1.5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
print(tokenizer.model_max_length)  # 512

For OpenAI models, the limit is documented in the API but you can also encode and count directly:.

import tiktoken

encoder = tiktoken.encoding_for_model("text-embedding-3-small")

def count_tokens(text: str) -> int:
    return len(encoder.encode(text))

def fits_in_window(text: str, max_tokens: int = 8191) -> bool:
    return count_tokens(text) <= max_tokens

A simple rule: set your chunk size to no more than 80% of the model's token window. This leaves room for the chunk overlap tokens and any metadata you prepend to chunks (e.g. document title or section header), which many retrieval patterns recommend.

Parent-child chunking

Standard chunking embeds and retrieves the same chunk that gets fed to the LLM. Parent-child chunking separates these two concerns: you embed small chunks for precise retrieval, but feed larger chunks to the LLM for richer context.

The intuition is straightforward. Small chunks (100–200 tokens) produce focused, specific embeddings , a tight vector that represents one idea clearly. Large chunks (500–1000 tokens) provide the surrounding context that an LLM needs to answer well. If you embed large chunks, the vector blurs across too many ideas and retrieval precision drops. If you feed small chunks to the LLM, it lacks the context to reason properly.

The architecture works like this:

• Child chunks are small, overlapping slices of the document. These are what you embed and store in your vector database.
• Parent chunks are the larger surrounding passages. These are stored in a regular key-value store or document store (not a vector database), indexed by a parent_id that each child chunk carries as metadata.
• At query time, the vector search finds the most relevant child chunks. You then look up their parent_id and retrieve the full parent chunk from the document store. The parent chunk , not the child , is what gets injected into the LLM's context window.

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=800)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)

vectorstore = Chroma(embedding_function=embeddings)
docstore = InMemoryStore()  # regular key-value store for parent chunks

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,    # stores child embeddings
    docstore=docstore,          # stores parent chunks
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

retriever.add_documents(docs)

# at query time: vector search finds child, retriever returns parent
results = retriever.invoke("What is LangGraph?")

The key insight is that the vector database and the document store serve different purposes and are never mixed. The vector database answers "which passage is most relevant?" The document store answers "give me the full context around that passage." The LLM only ever sees the parent.

This pattern is particularly effective for structured documents , technical documentation, legal texts, research papers , where a precise fact lives in one sentence but only makes sense in the context of the surrounding paragraph or section.

Text Similarity Concepts

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

sim = cosine_similarity([vec1], [vec2])[0][0]

Euclidean (L2) Distance
Measures the straight-line distance between two points in vector space. Lower = more similar. Unlike cosine, it is sensitive to vector magnitude, so it requires normalized embeddings to behave predictably. Some vector databases default to L2 , check your database's default metric and ensure it matches how your model was trained.

Dot Product
The raw inner product of two vectors. When vectors are L2-normalized (unit length), the dot product is mathematically equivalent to cosine similarity, so it becomes the fastest option , no normalization step at query time. This is why many embedding guides recommend normalizing embeddings: you get cosine semantics at dot-product speed.

Practical guidance: Use cosine similarity unless your vector database or model documentation recommends otherwise. Always normalize your embeddings ( normalize_embeddings=True in LangChain's HuggingFace wrapper) , it costs nothing and makes similarity scores more interpretable and consistent across different models and stores.

Embedding Dimensions

embeddings = OpenAIEmbeddings(model="text-embedding-3-small", dimensions=512)

Chunking Before Embedding

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFDirectoryLoader

loader = PyPDFDirectoryLoader("docs/")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,          # tokens ~ characters for English
    chunk_overlap=150,
    separators=["\n\n", "\n", ".", " ", ""]
)

chunks = splitter.split_documents(docs)
print(len(chunks))

from langchain_experimental.text_splitter import SemanticChunker

semantic_splitter = SemanticChunker(
    embeddings,                     # any embeddings model
    breakpoint_threshold_type="percentile",  # or "standard_deviation"
    breakpoint_threshold_amount=85
)

chunks = semantic_splitter.split_documents(docs)

Embedding Large Documents Best practices:

Embedding Quality Considerations

# Incremental update
new_chunks = splitter.split_documents(new_docs)
vectorstore.add_documents(new_chunks)

# Full rebuild (when changing model or chunking strategy)
vectorstore = Chroma.from_documents(
    documents=all_chunks,
    embedding=embeddings,
    collection_name="my_knowledge_base_v2",
    persist_directory="./chroma_db"
)

class FlexibleEmbeddings:
    def __init__(self, model_name: str = "text-embedding-3-small"):
        if "text-embedding" in model_name:
            self.embeddings = OpenAIEmbeddings(model=model_name)
        else:
            self.embeddings = HuggingFaceEmbeddings(model_name=model_name)
    
    def embed_documents(self, texts):
        return self.embeddings.embed_documents(texts)
    
    def embed_query(self, text):
        return self.embeddings.embed_query(text)

# Easy switching
embeddings = FlexibleEmbeddings("BAAI/bge-large-en-v1.5")