Bits, Nats, Perplexity, and BPB

The standard training loss for language models. For a model $q$ predicting tokens from a true distribution $p$ over vocabulary $V$:

$H_{\text{nats}}(p, q) = -\sum_{v \in V} p(v) \log q(v)$

where $\log$ is the natural logarithm (base $e$). This is what PyTorch's CrossEntropyLoss returns. The unit is nats (natural units of information).

A perfect model achieves $H_{\text{nats}} = H(p)$, the entropy of the true distribution. For natural language, this is typically 2 to 5 nats per token depending on the tokenizer and domain.

The same cross-entropy, but using $\log_2$ instead of $\ln$:

$H_{\text{bits}} = \frac{H_{\text{nats}}}{\ln 2} \approx \frac{H_{\text{nats}}}{0.6931}$

The unit is bits (binary digits of information). One nat $\approx$ 1.4427 bits. One bit $\approx$ 0.6931 nats. Information theory traditionally uses bits; machine learning traditionally uses nats.

The exponential of the cross-entropy:

$\text{PPL} = e^{H_{\text{nats}}} = 2^{H_{\text{bits}}}$

Perplexity has an intuitive interpretation: a perplexity of $k$ means the model is "as confused as if it were choosing uniformly among $k$ equally likely tokens at each step."

PPL = 1 means perfect prediction. PPL = $|V|$ means uniform random guessing over the vocabulary. For modern language models on standard benchmarks, PPL is typically 5 to 30.

Cross-entropy in bits, normalized by the number of UTF-8 bytes rather than the number of tokens:

$\text{BPB} = \frac{H_{\text{bits}} \times (\text{number of tokens})}{(\text{number of bytes in text})}$

BPB is tokenizer-independent: two models with different tokenizers can be compared directly because the normalization is by bytes, not tokens.

Typical values: 0.5 to 1.0 BPB for strong language models on English text. Shannon's estimate of English entropy is roughly 0.8 to 1.3 bits per character.