Coding outcomes from a real distribution P with a code sized for a model Q costs −Σ p log₂ q bits per symbol, which exceeds the entropy H(P) by D(P‖Q) ≥ 0, with equality only when Q matches P exactly, and D(P‖Q) ≠ D(Q‖P) in general.
Builds on Huffman coder. Taught in Bits & Surprise.
Drag the model Q's bars against a fixed reality P and watch the average code length and the overpay D(P‖Q) respond; set Q = P to hit zero, and swap the roles to see the divergence is not symmetric.
The claim above is what this widget demonstrates. Disagree with it?