Fix Invalid JSON from LLM API Responses

Your prompt says "respond with only JSON." The model responds with a stray sentence before the braces, a trailing comma, or the whole thing wrapped in a ```json code fence, and JSON.parse() throws SyntaxError: Unexpected token. If you're integrating an LLM API into a Node.js or Python backend, this is one of the most common failure modes you'll hit, and it usually shows up in production logs as an intermittent error rather than something you can reproduce on demand.

The fix isn't a more polite prompt. It's a different approach to how you ask for structured data and how you handle the response once it comes back.

Why LLM APIs Return Invalid JSON

Even when you write "return only valid JSON, no other text" in your system prompt, three things routinely break it:

Markdown wrapping. The model was trained on millions of examples of JSON inside ```json code fences, so it reaches for that habit even when you explicitly say not to.
Preamble text. Chat-tuned models like to explain themselves — "Here's the JSON you asked for:" — before the actual payload.
Truncation. If max_tokens is too low or the response gets cut off mid-stream, you end up with a JSON object that's syntactically incomplete, which parses as invalid no matter how well-formed your prompt was.
Malformed structure. Trailing commas, single quotes instead of double quotes, or an object key that isn't quoted at all — small deviations that a human reader would forgive but JSON.parse won't.

Prompt engineering can reduce how often this happens, but it can't get you to zero. If your integration depends on JSON.parse never throwing, you need a mechanism that constrains the output, not just a request that asks nicely.

Fix 1: Use Structured Output Instead of Prompt-and-Pray

Most major LLM providers now support a structured output mode that constrains generation to match a JSON Schema, rather than relying on the model to freeform its way into valid JSON. With OpenAI's API, that's the response_format parameter set to json_schema with strict: true:

import OpenAI from "openai";
const client = new OpenAI();

const schema = {
  name: "extract_ticket",
  strict: true,
  schema: {
    type: "object",
    properties: {
      title: { type: "string" },
      priority: { type: "string", enum: ["low", "medium", "high"] },
      tags: { type: "array", items: { type: "string" } },
    },
    required: ["title", "priority", "tags"],
    additionalProperties: false,
  },
};

const response = await client.chat.completions.create({
  model: "gpt-4o-2024-08-06",
  messages: [{ role: "user", content: rawTicketText }],
  response_format: { type: "json_schema", json_schema: schema },
});

const data = JSON.parse(response.choices[0].message.content);

With strict: true, the API guarantees the output conforms to the schema — no markdown fences, no preamble, no missing required fields. This eliminates the entire first category of failures (wrapping and preamble) at the source instead of trying to strip them out after the fact.

Fix 2: Validate the Shape, Not Just the Syntax

Valid JSON syntax doesn't mean valid data. A model can return {"priority": "urgent"} when your enum only allows "low" | "medium" | "high", and JSON.parse will happily accept it. Run every parsed response through a schema validator before your code touches it:

import { z } from "zod";

const TicketSchema = z.object({
  title: z.string().min(1),
  priority: z.enum(["low", "medium", "high"]),
  tags: z.array(z.string()),
});

function parseTicketResponse(raw) {
  const json = JSON.parse(raw);
  return TicketSchema.parse(json); // throws ZodError on shape mismatch
}

This turns a silent data-integrity bug (an unexpected "urgent" value flowing three layers deep into your database) into a loud, catchable error at the boundary where the LLM response enters your system — exactly where you want it to fail.

Fix 3: A Parse-and-Repair Fallback for Providers Without Strict Mode

Not every provider or model you call supports strict structured output. For those cases, wrap the parse in a fallback that strips the common wrapping patterns before giving up:

function safeJsonParse(raw) {
  try {
    return JSON.parse(raw);
  } catch {
    // Strip ```json ... ``` fences and any leading/trailing prose
    const match = raw.match(/\{[\s\S]*\}/);
    if (!match) throw new Error("No JSON object found in LLM response");
    return JSON.parse(match[0]);
  }
}

This regex-based repair handles the "explained itself first" and "wrapped in markdown" cases, which together cause the large majority of real-world parse failures. It won't fix truncated JSON — that's a max_tokens problem, not a parsing problem, and the real fix there is raising the token limit or asking for a smaller payload, not writing a more forgiving repair function.

Handling Streaming Responses

If you're streaming the response for latency reasons, don't attempt to JSON.parse partial chunks as they arrive — an incomplete object will always throw. Buffer the full stream first, then parse once, or use a dedicated streaming JSON parser (like partial-json on npm) that's built to tolerate incomplete input and return the best-effort object so far. Mixing raw chunk-by-chunk parsing with standard JSON.parse is the most common cause of "it works when I test it slowly but breaks in production" bug reports on this exact issue.

Common Mistakes

Relying only on prompt wording ("respond with JSON only") instead of the provider's actual structured-output parameter
Skipping schema validation after JSON.parse succeeds, so type and enum mismatches reach your database layer
Setting max_tokens too low for the expected response size, causing silent truncation that looks like a formatting bug
Retrying a failed parse with the exact same prompt and no repair logic, instead of falling back to a stricter format or a smaller schema

Best Practices for Reliable JSON from LLM APIs

Use structured output / JSON mode with a strict schema whenever the provider supports it — this is the real fix, not a workaround
Validate the parsed object against a schema (Zod, Joi, or similar) before using it anywhere downstream
Set max_tokens generously enough to cover your largest expected response, with headroom
Log the raw response alongside the parse failure so you can see exactly what broke, rather than just catching and swallowing the error
Keep a regex-based repair fallback only as a safety net for providers without strict mode — not as your primary strategy

Frequently Asked Questions

Why does JSON.parse fail even though I told the model to only return JSON? Because a plain-text instruction is a request, not a constraint — the model can still ignore it under normal sampling. Structured output modes (like OpenAI's json_schema with strict: true) enforce the format at the API level instead of relying on the model to comply.

Does structured output slow down the response or cost more? It doesn't meaningfully change cost, and any latency difference is negligible compared to the cost of a failed parse causing a retry or a dropped request in production.

What if my provider doesn't support structured output at all? Use the parse-and-repair fallback above, keep max_tokens generous, and add explicit formatting instructions with a one-shot example in the prompt — it won't be as reliable as strict mode, but it closes most of the gap.

Should I retry automatically when parsing fails? Yes, but retry with a stricter instruction or a smaller schema, and cap it at one or two attempts — an unbounded retry loop against a model that structurally can't produce the shape you want just burns tokens without fixing anything.

Key Takeaways

Invalid JSON from an LLM API is a constraint problem, not a wording problem. Prompting harder narrows the failure rate but never closes it, while structured output modes with a strict schema close it at the source. Pair that with schema validation on the parsed result, generous token limits, and a regex-based fallback for providers without strict mode, and JSON parsing stops being an intermittent production bug and becomes something you can actually rely on.