MATHEMATICS
Shannon’s 1948 breakthrough: two different jobs for a bit
Shannon’s 1948 theory separates compression from reliable transmission. Worked examples explain entropy, noisy channel capacity, and the assumptions behind both.
Research checked
Newest cited publication
A later vector recreation of Shannon’s communication system shows the information source, transmitter, noisy signal path, receiver, and destination. The article develops the probabilistic limits behind this architecture.
Wanderingstan and Stannered; cleaned by Reseletti / Wikimedia Commons. Unchanged original SVG. · Image source · Public domain (released by the creator)
A communication system faces two numerical questions before anyone chooses a cable or a radio antenna. How much uncertainty does its source generate, and how much can its channel reliably convey? Claude Shannon’s paper A Mathematical Theory of Communication, published in July and October 1948, gave these questions a common mathematical language. Compression removes predictable structure from a representation. Error control deliberately adds structure that helps a receiver recover a damaged representation. Understanding these different jobs explains why fewer transmitted bits and more transmitted bits can each improve a system. Its limits and existence arguments depend on explicit probabilistic assumptions.
Count possibilities, then account for their probabilities
Ralph Hartley’s July 1928 paper Transmission of Information developed a logarithmic measure based on possible symbol sequences. For an alphabet with m choices at each of n positions, allowing every combination gives m raised to n possible messages. Its base two logarithm is n log₂ m. The logarithm turns multiplication of independent choice counts into addition, making information budgets easier to combine. Two choices require one binary distinction; eight equally likely choices require three. The unit follows from the logarithm’s base, while the count depends on which messages are permitted.
Probabilities introduce another layer. If X takes values x with probabilities p(x), its entropy in bits is H(X) = Σ p(x) log₂(1/p(x)), with zero probability terms contributing zero. Each outcome receives a logarithmic surprise, and entropy averages those surprises. A fair binary outcome has entropy one. A deterministic outcome has entropy zero. For equally likely symbols this reduces to Hartley’s logarithmic count, but an uneven distribution can require a smaller average description. The relevant uncertainty belongs to a specified probability model; changing the model changes the calculation.
Five labels need fewer than three bits on average
Consider an independent source producing five labels A, B, C, D and E with probabilities one half, one quarter, one eighth, one sixteenth and one sixteenth. A fixed length binary representation needs three bits for every label because two bits distinguish only four possibilities. An alternative assigns A the string 0, B the string 10, C the string 110, D the string 1110, and E the string 1111. None of these complete strings begins another, so a receiver can parse consecutive labels without inserting separators. For example, 0101111 decodes uniquely as A, B, E.
The average length is one half times one, plus one quarter times two, plus one eighth times three, plus two lots of one sixteenth times four. The result is 1.875 bits per label. Computing entropy gives the same value because these probabilities are powers of one half. Compared with three bits, the expected saving is 37.5 percent. This toy model describes averages under fixed probabilities. Some labels take four bits, so the saving concerns an average over the stated distribution; it does not promise that every individual message becomes shorter.
An entropy rate needs a source model
The noiseless coding result makes entropy operational: under the appropriate source assumptions, coding sufficiently long blocks can approach the information rate. For a known independent finite alphabet source, variable length lossless block codes can make average bits per symbol approach H from above. Exact recovery still requires a uniquely decodable representation. A fixed length code that covers every possible block faces a different worst case counting constraint. It is therefore important to specify average length, allowed errors, and whether rare sequences receive longer descriptions before quoting a compression limit.
Dependence also changes the answer. Imagine a sequence that alternates A and B forever, with a fair initial choice deciding which label comes first. Each position alone has one bit of uncertainty. Yet a whole block of any positive length has only two possible realizations, so one bit identifies it once its length is known. The block entropy divided by block length tends to zero. The alternating sequence shows why a single symbol frequency table can overstate an entropy rate. Correlations and shared knowledge belong in the source description, alongside the alphabet and probabilities.
Capacity counts distinguishable messages after noise
For a finite alphabet memoryless channel, the output Y is produced from input X according to a specified conditional distribution, independently across uses once the inputs are given. The mutual information I(X;Y) equals H(Y) minus H(Y given X). Capacity is its maximum over permitted input distributions, measured here in bits per channel use. The maximization matters because different signaling frequencies can use the same physical channel differently. For rates strictly below capacity, sufficiently long block codes can drive the block decoding error probability arbitrarily close to zero under this model.
Take a binary channel that independently flips each transmitted bit with probability 0.08. For equally likely input bits, the output is also equally likely, giving H(Y) = 1. The remaining conditional uncertainty is the binary entropy h₂(0.08), approximately 0.402179 bits. Symmetry makes the uniform input optimal, so capacity is approximately 0.597821 bits per use. These numbers describe the specified ideal channel. The theorem does not guarantee zero errors for a finite code, bounded decoding effort, or a particular reliability exactly at capacity. Rates above capacity cannot achieve vanishing block error in this setting.
A small error correcting code is useful but limited
A concrete protection scheme sends 000 for a zero and 111 for a one, then decodes by majority vote. On the same independent flip channel, decoding fails when two or three of the three transmitted bits flip. Its failure probability is 3 × 0.08² × 0.92 + 0.08³ = 0.018176, or 1.8176 percent. That improves on the unprotected bit’s eight percent failure probability while spending three channel uses per information bit. Its rate is one third of a bit per use, below the calculated capacity.
Repeating this fixed three bit scheme across a longer message does not make its individual decisions progressively more reliable. Its per-decision error remains 1.8176 percent, and the chance that an entire long message contains some wrong decision increases. Shannon’s asymptotic coding statement instead permits different codes with growing block lengths. The distinction between one short working code and a sequence of increasingly reliable codes is essential. The existence of the latter does not turn the former into a capacity achieving design.
A bandwidth formula belongs to another precise model
Shannon’s January 1949 paper Communication in the Presence of Noise develops a geometric picture of signaling and analyzes transmission with noise. In the ideal additive white Gaussian noise model, with bandwidth B, average signal power P and noise power N in that band, capacity is B log₂(1 + P/N) bits per second. For an illustrative bandwidth of two thousand hertz and signal to noise power ratio fifteen, the formula gives eight thousand bits per second. The ratio is dimensionless. The bandwidth supplies the per-second unit. This is a calculation within that model, not a specification for an arbitrary radio.
The two channel examples have different assumptions: independent binary flips in one, additive Gaussian perturbations under a power and bandwidth constraint in the other. Neither formula can be carried over merely by renaming its parameters. Their common lesson is to separate the desired information rate from the resources and error probability needed to support it. Source statistics determine what can be compressed, while channel statistics constrain reliable protection. Shannon’s framework makes those questions testable without promising that an unknown source, changing noise, short delay or limited decoder will automatically satisfy the ideal theorem.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- A Mathematical Theory of CommunicationThe Bell System Technical Journal; corrected original-paper reprint hosted by Harvard MathematicsPublished Checked
- Transmission of InformationThe Bell System Technical Journal; original article scan retained by arsouyesPublished Checked
- Communication in the Presence of NoiseProceedings of the IRE; original 1949 work in a 1998 IEEE reprint hosted by IMJ-PRGPublished Checked
FOLLOW THE QUESTION
Explore mathematics