Mutual Information functions for C and MATLAB
BSD-3-CLAUSE License
v3.0.1 for C/C++ and MATLAB/Octave
MIToolbox contains a set of functions to calculate information theoretic quantities from data, such as the entropy and mutual information. The toolbox contains implementations of the most popular Shannon entropies, and also the lesser known Renyi entropy. The toolbox also provides implementations of the weighted entropy and weighted mutual information from "Information Theory with Application", S. Guiasu (1977). The toolbox only supports discrete distributions, as opposed to continuous. All real-valued numbers will be processed by x = floor(x).
These functions are targeted for use with feature selection algorithms rather than communication channels and so expect all the data to be available before execution and sample their own probability distributions from the data.
All functions expect the inputs to be vectors or matrices of doubles.
Functions contained:
Note: all functions are calculated in log base 2, so return units of "bits".
MIToolbox works on discrete inputs, and all continuous values must be discretised before use with MIToolbox. Real-valued inputs will be discretised with x = floor(x) to ensure compatibility. MIToolbox produces unreliable results when used with continuous inputs, runs slowly and uses much more memory than usual. The discrete inputs should have small cardinality, MIToolbox will treat values {1,10,100} the same way it treats {1,2,3} and the latter will be both faster and use less memory. This limitation is due to the difficulties in estimating information theoretic functions of continuous variables.
======
Examples:
>> y = [1 1 1 0 0]';
>> x = [1 0 1 1 0]';
>> mi(x,y) %% mutual information I(X;Y)
ans =
0.0200
>> h(x) %% entropy H(X)
ans =
0.9710
>> condh(x,y) %% conditional entropy H(X|Y)
ans =
0.9510
>> h( [x,y] ) %% joint entropy H(X,Y)
ans =
1.9219
>> joint([x,y]) %% joint random variable XY
ans =
1
2
1
3
4
======
All code is licensed under the 3-clause BSD license.
Compilation instructions:
CompileMIToolbox.m
from the matlab
directory.make x86
or make x64
for 32-bit or 64-bit versions respectively.sudo make install
to install MIToolbox into /usr/local/lib
and /usr/local/include
mingw/bin
, mingw/msys/bin
make x64_win
to compile a 64-bit Windows dll.Update History