FREE Ideas & theory LESSON · Information has a unit
Measure surprise
Entropy predicts compression
Predictable messages need fewer bits.
A source that repeats the same symbol has almost no surprise. A source where many symbols are equally likely has high entropy and needs more space on average.
Compression removes predictability; it cannot remove genuine uncertainty.