ENRU

JSON Lines (NDJSON): what it is and how to read it

Updated

JSON Lines is a text file where every line is a complete JSON value. The file as a whole is not one JSON document, so a regular parser reads the first line and stops at the second. That stop is Python’s “Extra data”.

What a JSON Lines file looks like

{"id":1,"event":"login","user":"ann"}
{"id":2,"event":"view","user":"bob"}
{"id":3,"event":"logout","user":"ann"}
  • UTF-8 text, one JSON value per line, usually an object. A value never spans lines: a line break inside a string is written as \n.
  • Lines end with \n; \r\n also works, because whitespace around a value is ignored. End the last line with \n too, so the next append starts on a new line.
  • Blank lines are not records. Many readers skip them, some fail: don’t write them.
  • Extensions .jsonl and .ndjson. JSON Lines and NDJSON (newline-delimited JSON) are two specs for the same format; application/x-ndjson is the media type you meet most, for example in the Elasticsearch bulk API.

Why json.load fails with “Extra data”

>>> json.load(open("events.jsonl"))
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 38)

RFC 8259 defines a JSON text as one value with optional whitespace around it. The parser reads the first object, finds { at the start of line 2 and refuses to guess. Every strict parser says the same in its own words:

ParserMessage for the file aboveChecked
Python json.loadExtra data: line 2 column 1 (char 38)run, Python 3.9.6
V8 JSON.parse (Node.js, Chrome)Unexpected non-whitespace character after JSON at position 38 (line 2 column 1)run, Node 25.1.0
Firefox JSON.parseunexpected non-whitespace character after JSON datasource
Go json.Unmarshalinvalid character '{' after top-level valuerun, Go 1.25.3
C# System.Text.Json'{' is invalid after a single JSON value. Expected end of data.source
jqno error: jq reads a stream of valuesrun, jq 1.7.1
Our ValidatorExtra data after JSON, line 2, column 1run

If the file really is JSON Lines, read it line by line, as below. If it was meant to be one document, something else follows the value: a second pasted object, a stray bracket, a log line. Extra data after JSON covers those cases; try it in the Validator.

Read and write JSON Lines in Python

import json

def read_jsonl(path):
    with open(path, encoding="utf-8") as f:
        for n, line in enumerate(f, start=1):
            if not line.strip():
                continue
            try:
                yield json.loads(line)
            except json.JSONDecodeError as e:
                raise ValueError(f"{path}:{n}: {e}") from None

for record in read_jsonl("events.jsonl"):
    print(record["id"], record["event"])

with open("out.jsonl", "w", encoding="utf-8") as f:
    for record in records:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

The generator holds one line in memory, so a file of any size works, and an error names the line: events.jsonl:3: Expecting property name enclosed in double quotes: line 1 column 21 (char 20). json.dumps without indent never writes a raw line break, so each record stays on its line. pandas reads the format with pd.read_json(path, lines=True).

Read JSON Lines in JavaScript and Node.js

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const lines = createInterface({ input: createReadStream('events.jsonl'), crlfDelay: Infinity });
let n = 0;
for await (const line of lines) {
  n++;
  if (!line.trim()) continue;
  const record = JSON.parse(line);
  console.log(record.id, record.event);
}

// A small file already in memory:
const records = text.split('\n').filter((l) => l.trim()).map((l) => JSON.parse(l));
// Writing:
const out = records.map((r) => JSON.stringify(r)).join('\n') + '\n';

readline streams the file, so memory stays flat. Keep the counter n to report which line failed: V8’s message counts the position inside the line, not in the file.

Read JSON Lines in Go

dec := json.NewDecoder(f)
for {
	var e Event
	if err := dec.Decode(&e); err == io.EOF {
		break
	} else if err != nil {
		return err
	}
	fmt.Println(e.ID, e.Event)
}

json.Decoder reads value after value and treats line breaks as whitespace. To report line numbers, use bufio.Scanner with json.Unmarshal per line, but raise its limit first: the default buffer is 64 KiB, and a longer line stops the scan with bufio.Scanner: token too long. Call sc.Buffer(make([]byte, 0, 64*1024), 10*1024*1024) before the loop. Writing is one call per record: json.NewEncoder(w).Encode(v) adds the newline itself.

JSON Lines with jq

jq -c 'select(.user == "ann")' events.jsonl   # filter, one record per line
jq -s '.' events.jsonl > events.json           # JSON Lines to one array
jq -c '.[]' events.json > events.jsonl         # array back to JSON Lines
jq -s 'length' events.jsonl                    # count records

jq reads a stream of values, so JSON Lines needs no flag. -c prints each result on one line, -s (slurp) collects everything into an array. A broken line stops jq with its position in the file: jq: parse error: Expected another key-value pair at line 3, column 21.

When to use JSON Lines, and when not

  • Logs and event streams. Appending a record is writing one line; nothing earlier is rewritten, and a crash costs at most the last line.
  • Large exports. A reader holds one record at a time, and head, tail, split, wc -l and grep work on records without a JSON parser.
  • Datasets and batch APIs. Training data and batch job files often come as .jsonl; BigQuery calls the format newline-delimited JSON.
  • Bulk loading. Elasticsearch’s _bulk endpoint takes NDJSON: an action line, then a document line.

Use one JSON document instead for configuration, for API responses read in one piece, and for anything a person edits by hand. Our formatter and Validator check one document: to look at a JSON Lines file there, paste a single line, or turn the file into an array with jq -s.

JSON Lines FAQ

Is a JSON Lines file valid JSON?

No. Each line is valid JSON; the file as a whole is not, which is why json.load and JSON.parse stop at line 2.

What’s the difference between JSON Lines and NDJSON?

In practice none: both mean one JSON value per line, separated by \n. They are two specs written for the same idea, with the extensions .jsonl and .ndjson.

How do I convert JSON Lines to a JSON array?

jq -s . file.jsonl > file.json. In Python: json.dump(list(read_jsonl(path)), f). Back from an array: jq -c '.[]' file.json.

Can one record span several lines?

Not in JSON Lines. jq and Go’s json.Decoder would still read pretty printed records, but line-based tools would not, and the file would stop being JSON Lines.