JSON Lines (NDJSON): what it is and how to read it
Updated
JSON Lines is a text file where every line is a complete JSON value. The file as a whole is not one JSON document, so a regular parser reads the first line and stops at the second. That stop is Python’s “Extra data”.
What a JSON Lines file looks like
{"id":1,"event":"login","user":"ann"}
{"id":2,"event":"view","user":"bob"}
{"id":3,"event":"logout","user":"ann"}- UTF-8 text, one JSON value per line, usually an object. A value never spans lines: a line break inside a string is written as
\n. - Lines end with
\n;\r\nalso works, because whitespace around a value is ignored. End the last line with\ntoo, so the next append starts on a new line. - Blank lines are not records. Many readers skip them, some fail: don’t write them.
- Extensions
.jsonland.ndjson. JSON Lines and NDJSON (newline-delimited JSON) are two specs for the same format;application/x-ndjsonis the media type you meet most, for example in the Elasticsearch bulk API.
Why json.load fails with “Extra data”
>>> json.load(open("events.jsonl"))
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 38)RFC 8259 defines a JSON text as one value with optional whitespace around it. The parser reads the first object, finds { at the start of line 2 and refuses to guess. Every strict parser says the same in its own words:
| Parser | Message for the file above | Checked |
|---|---|---|
Python json.load | Extra data: line 2 column 1 (char 38) | run, Python 3.9.6 |
V8 JSON.parse (Node.js, Chrome) | Unexpected non-whitespace character after JSON at position 38 (line 2 column 1) | run, Node 25.1.0 |
Firefox JSON.parse | unexpected non-whitespace character after JSON data | source |
Go json.Unmarshal | invalid character '{' after top-level value | run, Go 1.25.3 |
| C# System.Text.Json | '{' is invalid after a single JSON value. Expected end of data. | source |
| jq | no error: jq reads a stream of values | run, jq 1.7.1 |
| Our Validator | Extra data after JSON, line 2, column 1 | run |
If the file really is JSON Lines, read it line by line, as below. If it was meant to be one document, something else follows the value: a second pasted object, a stray bracket, a log line. Extra data after JSON covers those cases; try it in the Validator.
Read and write JSON Lines in Python
import json
def read_jsonl(path):
with open(path, encoding="utf-8") as f:
for n, line in enumerate(f, start=1):
if not line.strip():
continue
try:
yield json.loads(line)
except json.JSONDecodeError as e:
raise ValueError(f"{path}:{n}: {e}") from None
for record in read_jsonl("events.jsonl"):
print(record["id"], record["event"])
with open("out.jsonl", "w", encoding="utf-8") as f:
for record in records:
f.write(json.dumps(record, ensure_ascii=False) + "\n")The generator holds one line in memory, so a file of any size works, and an error names the line: events.jsonl:3: Expecting property name enclosed in double quotes: line 1 column 21 (char 20). json.dumps without indent never writes a raw line break, so each record stays on its line. pandas reads the format with pd.read_json(path, lines=True).
Read JSON Lines in JavaScript and Node.js
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';
const lines = createInterface({ input: createReadStream('events.jsonl'), crlfDelay: Infinity });
let n = 0;
for await (const line of lines) {
n++;
if (!line.trim()) continue;
const record = JSON.parse(line);
console.log(record.id, record.event);
}
// A small file already in memory:
const records = text.split('\n').filter((l) => l.trim()).map((l) => JSON.parse(l));
// Writing:
const out = records.map((r) => JSON.stringify(r)).join('\n') + '\n';readline streams the file, so memory stays flat. Keep the counter n to report which line failed: V8’s message counts the position inside the line, not in the file.
Read JSON Lines in Go
dec := json.NewDecoder(f)
for {
var e Event
if err := dec.Decode(&e); err == io.EOF {
break
} else if err != nil {
return err
}
fmt.Println(e.ID, e.Event)
}json.Decoder reads value after value and treats line breaks as whitespace. To report line numbers, use bufio.Scanner with json.Unmarshal per line, but raise its limit first: the default buffer is 64 KiB, and a longer line stops the scan with bufio.Scanner: token too long. Call sc.Buffer(make([]byte, 0, 64*1024), 10*1024*1024) before the loop. Writing is one call per record: json.NewEncoder(w).Encode(v) adds the newline itself.
JSON Lines with jq
jq -c 'select(.user == "ann")' events.jsonl # filter, one record per line
jq -s '.' events.jsonl > events.json # JSON Lines to one array
jq -c '.[]' events.json > events.jsonl # array back to JSON Lines
jq -s 'length' events.jsonl # count recordsjq reads a stream of values, so JSON Lines needs no flag. -c prints each result on one line, -s (slurp) collects everything into an array. A broken line stops jq with its position in the file: jq: parse error: Expected another key-value pair at line 3, column 21.
When to use JSON Lines, and when not
- Logs and event streams. Appending a record is writing one line; nothing earlier is rewritten, and a crash costs at most the last line.
- Large exports. A reader holds one record at a time, and
head,tail,split,wc -landgrepwork on records without a JSON parser. - Datasets and batch APIs. Training data and batch job files often come as
.jsonl; BigQuery calls the format newline-delimited JSON. - Bulk loading. Elasticsearch’s
_bulkendpoint takes NDJSON: an action line, then a document line.
Use one JSON document instead for configuration, for API responses read in one piece, and for anything a person edits by hand. Our formatter and Validator check one document: to look at a JSON Lines file there, paste a single line, or turn the file into an array with jq -s.
JSON Lines FAQ
Is a JSON Lines file valid JSON?
No. Each line is valid JSON; the file as a whole is not, which is why json.load and JSON.parse stop at line 2.
What’s the difference between JSON Lines and NDJSON?
In practice none: both mean one JSON value per line, separated by \n. They are two specs written for the same idea, with the extensions .jsonl and .ndjson.
How do I convert JSON Lines to a JSON array?
jq -s . file.jsonl > file.json. In Python: json.dump(list(read_jsonl(path)), f). Back from an array: jq -c '.[]' file.json.
Can one record span several lines?
Not in JSON Lines. jq and Go’s json.Decoder would still read pretty printed records, but line-based tools would not, and the file would stop being JSON Lines.