Node.js RAG Pipeline: pgvector + OpenAI

Most "add AI to our app" projects start the same way: bolt an OpenAI call onto an existing endpoint, ship a demo, and then watch it fall apart the moment someone asks a question the model has no way of actually answering. A Node.js RAG pipeline with pgvector fixes that specific failure mode by giving the model real, current data to reason over instead of relying on whatever it memorized during training. If you already run Postgres for your application data, pgvector lets you add retrieval-augmented generation without introducing a separate vector database, a new ops surface, or a second point of failure.

This is the architecture I reach for on client projects that need grounded answers — support bots that must quote the actual docs, internal tools that search a knowledge base, or product search that understands intent rather than just keywords.

What RAG actually solves

A large language model answers from its training data plus whatever you put in the prompt. Retrieval-augmented generation is the practice of fetching the right few paragraphs from your own content and injecting them into the prompt before the model responds. The model still generates the final text, but it's now grounded in something real and current, which cuts hallucination rates dramatically and lets you cite sources.

The retrieval step needs a way to find "similar" content to a user's question, and that's where vector embeddings and pgvector come in. Instead of keyword matching, you convert both your documents and the incoming query into high-dimensional vectors, then find the nearest neighbors by cosine or Euclidean distance.

Why pgvector instead of a dedicated vector database

Pinecone, Weaviate, and Qdrant are all solid, but they add a service you have to run, monitor, and pay for separately. If your data already lives in Postgres, pgvector — a Postgres extension — lets you store embeddings as a native column type and query them with regular SQL, joined against your existing tables. For most applications under a few million vectors, an IVFFlat or HNSW index on pgvector performs well enough that a separate vector store is unnecessary engineering overhead.

The tradeoff: pgvector asks more of your Postgres instance in terms of RAM and index tuning at very large scale. For most Node.js backends, that ceiling is well above what you'll actually hit.

Setting up the schema

Enable the extension and store embeddings alongside your document text:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
  id SERIAL PRIMARY KEY,
  document_id INTEGER REFERENCES documents(id),
  content TEXT NOT NULL,
  embedding VECTOR(1536),
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON document_chunks
  USING hnsw (embedding vector_cosine_ops);

1536 matches OpenAI's text-embedding-3-small output dimension — adjust it if you use a different embedding model.

Building the pipeline in Node.js

The pipeline has two phases: ingesting and embedding your content once, then retrieving and generating on every user query.

import OpenAI from "openai";
import { Pool } from "pg";

const openai = new OpenAI();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function embed(text) {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

async function ingestChunk(documentId, content) {
  const embedding = await embed(content);
  await pool.query(
    `INSERT INTO document_chunks (document_id, content, embedding)
     VALUES ($1, $2, $3)`,
    [documentId, content, JSON.stringify(embedding)]
  );
}

async function retrieve(query, limit = 5) {
  const queryEmbedding = await embed(query);
  const { rows } = await pool.query(
    `SELECT content, 1 - (embedding <=> $1) AS similarity
     FROM document_chunks
     ORDER BY embedding <=> $1
     LIMIT $2`,
    [JSON.stringify(queryEmbedding), limit]
  );
  return rows;
}

async function answer(query) {
  const chunks = await retrieve(query);
  const context = chunks.map((c) => c.content).join("\n---\n");

  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content:
          "Answer only using the provided context. If the context doesn't contain the answer, say so.",
      },
      { role: "user", content: `Context:\n${context}\n\nQuestion: ${query}` },
    ],
  });

  return completion.choices[0].message.content;
}

The <=> operator is pgvector's cosine distance operator; 1 - distance converts it to a similarity score you can log or expose for debugging relevance.

Chunking strategy matters more than model choice

Teams usually spend their optimization time on the embedding model or the LLM, but chunk size and overlap decide whether retrieval actually works:

200–500 tokens per chunk is a reasonable default for prose documentation — small enough to be specific, large enough to keep context intact.
Overlap chunks by 10–20% so an answer that straddles a chunk boundary doesn't get cut in half.
Split on semantic boundaries (headings, paragraphs) rather than a fixed character count wherever your source format allows it.
Store the source document and section reference with every chunk so you can cite it back to the user, not just answer from it silently.

Common mistakes

A few issues show up in almost every pgvector implementation I've reviewed:

No index on the embedding column. A sequential scan over tens of thousands of vectors is slow enough to notice; add the HNSW or IVFFlat index before you ship, not after a support ticket.
Re-embedding on every deploy. Embeddings are expensive to regenerate at scale — version your chunking logic and only re-embed documents that actually changed.
No relevance threshold. Returning the "top 5" results even when none of them are actually relevant produces confident-sounding wrong answers. Check the similarity score and let the model say "I don't know" below a threshold.
Ignoring token limits when building context. Concatenating five long chunks into a prompt without checking token count is a common source of truncated or failed completions in production.

Frequently Asked Questions

Does pgvector scale to production traffic? Yes, for the vast majority of applications. With an HNSW index, query latency stays in the low tens of milliseconds up to several million vectors on reasonably sized Postgres hardware. Very large-scale, high-QPS vector search (hundreds of millions of vectors) is where a dedicated vector database starts to pull ahead.

Which embedding model should I use? OpenAI's text-embedding-3-small is a solid, inexpensive default for most RAG applications. text-embedding-3-large improves retrieval quality at a higher cost per call — worth testing if relevance is borderline.

How do I keep the vector index updated as documents change? Re-embed and re-insert only the chunks belonging to the changed document, keyed by document_id, rather than rebuilding the whole table. A background job triggered on document save works well for most content-management workflows.

Can I combine vector search with keyword search? Yes — hybrid search that combines pgvector similarity with Postgres full-text search (tsvector) often outperforms either approach alone, especially for queries containing exact product names or error codes that embeddings alone can miss.

Conclusion

A Node.js RAG pipeline built on pgvector gets you grounded, source-backed AI answers without adding a new database service to your stack. Start with a proper HNSW index and a deliberate chunking strategy — those two decisions affect retrieval quality far more than which embedding model you pick, and getting them right the first time will save you a painful re-indexing pass later.