Definition
Information
In information theory, information quantifies the uncertainty removed by learning an outcome. Let be a discrete random variable with probability mass function . Observing an outcome with conveys
units of information. Less probable outcomes convey more information because they eliminate more possibilities. The base determines the unit: gives bits, while gives nats. The expected information conveyed by is its entropy.