Information entropy

Information Entropy is a concept from information theory. It tells how much information there is in an event. In general, the more uncertain or random the event is, the more information it will contain. The concept of information entropy was created by a mathematician. He was named Claude Elwood Shannon.

It has applications in many areas, including lossless data compression, statistical inference, cryptography and recently in other disciplines as biology, physics or machine learning.

Example

Let's look at an example. If someone is told something they already know, the information they get is very small. It will be pointless for them to be told something they already know. This information would have very low entropy.

If they were told about something they knew little about, they would get much new information. This information would be very valuable to them. They would learn something. This information would have high entropy.

Other pages

• Entropy encoding
• Kolmogorov-Sinai entropy in dynamical systems
• Theil index
• Thermodynamic entropy in Thermodynamics
• Information Entropy on English Wikipedia

Other websites

• Information is not entropy, ! - a discussion of the use of the terms "information" and "entropy".
• I'm Confused: How Could Information Equal Entropy? - a similar discussion on the bionet.info-theory FAQ.
• Java "entropy pool" for cryptographically-secure unguessable random numbers
• Description of information entropy from "Tools for Thought" by Howard Rheingold
• Entropy an interdisciplinary journal on all aspect of the entropy concept. Open access.
• An Intuitive Guide to the Concept of Entropy Arising in Various Sectors of Science – a wikibook on the interpretation of the concept of entropy.
• Calculator for Shannon entropy estimation and interpretation