Statistics and Information Theory

This is a Python library containing classes for Statistic and Information Theoretical computations. It also contains some code from Peter Norvig, AI: A Modern Appproach : http://aima.cs.berkeley.edu/python/utils.html

class statistics.Distribution(data, base=2)

A distribution can be created over a FrequencyList or a plain dictionary with numeric values. It will be normalized automatically.

entropy(base=None)

Compute the entropy of the distribution (base e)

information(type)

Computes the information content of the specified type: -log_e(p(X))

items()

Returns an unranked list of (type, prob) pairs. Use this only if you are not interested in the order.

maxentropy(base=2)

Compute the maximum entropy of the distribution: log_e(N)

mode()

Returns the type that occurs the most frequently in the probability distribution

output(delimiter='t')

poslog(type)

alias for information content

class statistics.FrequencyList(tokens=None, casesensitive=True)

append(tokens)

Add a list of tokens to the frequencylist. This method will count them for you.

count(type, amount=1)

Count a certain type. The counter will increase by the amount specified (defaults to one)

items()

Returns an unranked list of (type, count) pairs. Use this only if you are not interested in the order.

load(filename)

Load a frequency list from file (in the format produced by the save method)

mode()

Returns the type that occurs the most frequently in the frequency list

output(delimiter='t')

Print a representation of the frequency list

p(type)

Returns the probability (relative frequency) of the token

save(filename)

Save a frequency list to file, can be loaded later using the load method

sum()

Returns the total amount of tokens

tokens()

Returns the total amount of tokens

typetokenratio()

Computes the type/token ratio

statistics.dotproduct(X, Y)

Return the sum of the element-wise product of vectors x and y. >>> dotproduct([1, 2, 3], [1000, 100, 10]) 1230

statistics.histogram(values, mode=0, bin_function=None)

Return a list of (value, count) pairs, summarizing the input values. Sorted by increasing value, or if mode=1, by decreasing count. If bin_function is given, map it over values first.

statistics.levenshtein(s1, s2)

Computes the levenshtein distance between to strings. From: http://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Python

statistics.log2(x)

Base 2 logarithm. >>> log2(1024) 10.0

statistics.mean(values)

Return the arithmetic average of the values.

statistics.median(values)

Return the middle value, when the values are sorted. If there are an odd number of elements, try to average the middle two. If they can’t be averaged (e.g. they are strings), choose one at random. >>> median([10, 100, 11]) 11 >>> median([1, 2, 3, 4]) 2.5

statistics.mode(values)

Return the most common value in the list of values. >>> mode([1, 2, 3, 2]) 2

statistics.normalize(numbers, total=1.0)

Multiply each number by a constant such that the sum is 1.0 (or total). >>> normalize([1,2,1]) [0.25, 0.5, 0.25]

statistics.product(seq)

Return the product of a sequence of numerical values. >>> product([1,2,6]) 12

statistics.stddev(values, meanval=None)

The standard deviation of a set of values. Pass in the mean if you already know it.

statistics.vector_add(a, b)

Component-wise addition of two vectors. >>> vector_add((0, 1), (8, 9)) (8, 10)