Monthly Archives: June 2013

Really simple multi-processing in Python – and then linking to MongoDB

Multiprocessing in Python is really easy! You can spawn processes by using the Pool() class.

from multiprocessing import Pool

def add_one(to_what):
    """A trivial function to increment an integer"""
    return to_what + 1

pool = Pool(processes=4) #spawn 4 processes
numbers = [1,2,3,4] #this is our test data
result = pool.map(add_one,numbers) #send the function and 1 number to each
pool.close() #kill the processes
print result #will return [2,3,4,5]

from multiprocessing import Pool

def add_one(to_what):

"""A trivial function to increment an integer"""

return to_what + 1

pool = Pool(processes=4) #spawn 4 processes

numbers = [1,2,3,4] #this is our test data

result = pool.map(add_one,numbers) #send the function and 1 number to each

pool.close() #kill the processes

print result #will return [2,3,4,5]

Here, we spawn 4 processes, and use the map() function to send a number to each of them.

This is a trivial example, but gets much more powerful when each process does something like making a remote connection:

from multiprocessing import Pool
from pymongo import Connection

def how_many(server_number):
    """Returns how many documents in the collection"""
    c = Connection("192.168.0." + server_number) #connects to remote DB
    return c.MyDB.MyCollection.count()

pool = Pool(processes=4)
servers = [1,2,3,4]
result = pool.map(how_many,servers)
pool.close()

print "You have {0} docs across all MongoDB servers!".format(sum(result))

from multiprocessing import Pool

from pymongo import Connection

def how_many(server_number):

"""Returns how many documents in the collection"""

c = Connection("192.168.0." + server_number) #connects to remote DB

return c.MyDB.MyCollection.count()

pool = Pool(processes=4)

servers = [1,2,3,4]

result = pool.map(how_many,servers)

pool.close()

print "You have {0} docs across all MongoDB servers!".format(sum(result))

Markov Transition Matrix Equilibrium made simple in Python

Leave a reply

As a refresher on Markov Models, I’ve been watching Professor Scott E. Page’s excellent videos on YouTube. He does all the computation by hand – and now I’ve written some code to perform it faster.

In video 2, we are given a transition matrix of 1 hypothetical student in a classroom transitioning between alertness and boredom:

	Alert: t	Bored: t
Alert: t+1	0.8	0.25
Bored: t+1	0.2	0.75

This can be represented in Python as:

matrix  = [[0.8, 0.25],
           [0.2, 0.75]]

1 2	matrix = [[0.8, 0.25], [0.2, 0.75]]

The vector of students is again, a simple list:

students_vector = [1,0]

1	students_vector = [1,0]

Let’s calculate one stage:

def markov_stage(matrix, vector):
    """Calculates one iteration of the markov process"""
    v1 = [] # blank list to store the next stage
    for x in xrange(len(matrix)): #for each row of the matrix
        #multiply each element of the row by each element of the vector and sum
        #the "zip" function is great for this
        v1.append(sum([a*b for a,b in zip(matrix[x],vector)]))
    print v1
    return v1

def markov_stage(matrix, vector):

"""Calculates one iteration of the markov process"""

v1 = [] # blank list to store the next stage

for x in xrange(len(matrix)): #for each row of the matrix

#multiply each element of the row by each element of the vector and sum

#the "zip" function is great for this

v1.append(sum([a*b for a,b in zip(matrix[x],vector)]))

print v1

return v1

Calling this produces:

>>> student_vector = markov_stage(student_vector)
[0.8, 0.2]
>>>
>>> student_vector = markov_stage(student_vector)
[0.6900000000000002, 0.31000000000000005]

>>> student_vector = markov_stage(student_vector)

[0.8, 0.2]

>>>

>>> student_vector = markov_stage(student_vector)

[0.6900000000000002, 0.31000000000000005]

Or, let’s try to find the equilibrium state by looping markov_stage until the values basically stop changing (to a certain decimal place accuracy):

def markov_equilibrium(matrix, vector, acc):
    """Tries to find the equilibrium of a markov process.
       Accepts a matrix [i,j], a vector [v1,v2,vx..] and
       accuracy to any number of decimal points"""

    #"its" = iterations
    #We must have at least 2 iterations performed in order to judge
    #whether it looks like it is stabilising (basically just comparing the two)
    its = [markov_stage(matrix, vector)] #input first iteration
    its.append(markov_stage(matrix, its[-1])) #append another

    #now loop with the condition that, when rounded, the vector is not the same
    #as the one before
    while([round(x,acc) for x in its[-1]] != [round(x,acc) for x in its[-2]]):
        its.append(markov_stage(matrix, its[-1])) #we can call the function
                                                  #from before

    result = "Markov process took {0} stages to reach {1} d.p. equilibrium"
    print result.format(len(its), acc) #output a nice result

def markov_equilibrium(matrix, vector, acc):

"""Tries to find the equilibrium of a markov process.

Accepts a matrix [i,j], a vector [v1,v2,vx..] and

accuracy to any number of decimal points"""

#"its" = iterations

#We must have at least 2 iterations performed in order to judge

#whether it looks like it is stabilising (basically just comparing the two)

its = [markov_stage(matrix, vector)] #input first iteration

its.append(markov_stage(matrix, its[-1])) #append another

#now loop with the condition that, when rounded, the vector is not the same

#as the one before

while([round(x,acc) for x in its[-1]] != [round(x,acc) for x in its[-2]]):

its.append(markov_stage(matrix, its[-1])) #we can call the function

#from before

result = "Markov process took {0} stages to reach {1} d.p. equilibrium"

print result.format(len(its), acc) #output a nice result

This produces, with an accuracy set at 2 decimal places:

>>> markov_equilibrium(matrix,student_vector,2)
[0.8, 0.2]
[0.6900000000000002, 0.31000000000000005]
[0.6295000000000002, 0.37050000000000005]
[0.5962250000000002, 0.4037750000000001]
[0.5779237500000002, 0.42207625000000015]
[0.5678580625000003, 0.4321419375000002]
[0.5623219343750003, 0.4376780656250002]
[0.5592770639062503, 0.44072293609375024]
Markov process took 8 stages to reach 2 d.p. equilibrium

>>> markov_equilibrium(matrix,student_vector,2)

[0.8, 0.2]

[0.6900000000000002, 0.31000000000000005]

[0.6295000000000002, 0.37050000000000005]

[0.5962250000000002, 0.4037750000000001]

[0.5779237500000002, 0.42207625000000015]

[0.5678580625000003, 0.4321419375000002]

[0.5623219343750003, 0.4376780656250002]

[0.5592770639062503, 0.44072293609375024]

Markov process took 8 stages to reach 2 d.p. equilibrium

I probably could have written this in Numpy – which would calculate faster using less memory (and probably has built-in functions for the vector-matrix row multiplication), but it was fun just doing this. I’ll try and extend the markov_equilibrium to give some more detailed stats such as the “churn” as mentioned by Prof. Page.

Custom DefaultDicts with Lambdas in Python for creating detailed frequency tables, or anything else

Leave a reply

In Python, there is the dictionary datatype. This is basically a look-up table:

d = {}
d['hello'] = 'bonjour' # a string
d['world'] = ['monde', u'世界']　# a list of strings

d = {}

d['hello'] = 'bonjour' # a string

d['world'] = ['monde', u'世界']　# a list of strings

Let’s try and create a frequency table for words though:

def make_frequency_table(list_of_things):
    table = {}
    for item in list_of_things:
        try:
            table[item] += 1
        except KeyError:
            table[item] = 1
    return table

def make_frequency_table(list_of_things):

table = {}

for item in list_of_things:

try:

table[item] += 1

except KeyError:

table[item] = 1

return table

This bad code will eventually return a dictionary with entries like:

thing: how_many_times_it_occurred

However, we have to do this try-except statement in case the key doesn’t exist. Defaultdicts get rid of this stage by returning a blank entry (0, empty string, empty list) instead, which is really awesome!

from collections import defaultdict

def make_frequency_table(list_of_things):
    table = defaultdict(int)
    for item in list_of_things:
        table[item] += 1
    return table

from collections import defaultdict

def make_frequency_table(list_of_things):

table = defaultdict(int)

for item in list_of_things:

table[item] += 1

return table

If the key didn’t already exist in our look-up table, then the defaultdict returns an <int> to write the new value! This defaultdict(int) could be replaced with defaultdict(list) or any data type.

And now to the crux of the post! We can replace this variable type with a lambda instead, like this:

table = defaultdict(lambda: {'frequency':0, 'last_seen_at':0 })

1	table = defaultdict(lambda: {'frequency':0, 'last_seen_at':0 })

Now, when the key doesn’t exist, the dictionary will create a new dictionary within! So we can bring another metric into our analysis:

from collections import defaultdict

def make_frequency_table(list_of_things):
    table = defaultdict(lambda: {'frequency':0, 'last_seen_at':0 })
    for position, item in enumerate(list_of_things):
        table[item]['frequency'] += 1
        table[item]['last_seen_at'] = position
    return table

from collections import defaultdict

def make_frequency_table(list_of_things):

table = defaultdict(lambda: {'frequency':0, 'last_seen_at':0 })

for position, item in enumerate(list_of_things):

table[item]['frequency'] += 1

table[item]['last_seen_at'] = position

return table

Now our function will return a dictionary that not only lets you know how many times something occurred, but also when it last occurred! Try it out with the following data:

dataset = [1,1,1,1,2,2,2,2,2,5,5,5,1,4,5,6,8,9]

1	dataset = [1,1,1,1,2,2,2,2,2,5,5,5,1,4,5,6,8,9]

Two quick fixes to make your web browsing more anonymous

Leave a reply

Two very easy ways to make your browsing more secure would be to install the following two extensions in Google Chrome:

Ghostery – blocks tracking scripts
AdBlock – blocks advertising

Why?

These block (usually third-party) components which let companies log your browsing data. Check out the Collusion extension to see who knows what. This tries to stop companies easily doing things like (allegedly) increasing airfares and hotel prices based on your browsing history.
A nice side effect is that pages may load faster due their being less third-party components to load. Often, sites will add these components to the head of the page, before any other page elements to make sure that the advertising is given a priority to load over the actual page content.

Caveats:

You can still be easily monitored on an ISP level, by a toolbar or if the website is selling off internally logged data.
Blocking components using these may “break” some websites by removing essential page-formatting style-sheets and scripts.
Blocking advertising from your favorite websites may reduce their advertising income and not let them be rewarded for their traffic/efforts.

Hypocrisy:

I use “statcounter” on this website, which lets me see how many page views there are. StatCounter also aggregates all of its clients data into a very interesting public global stats page. In my opinion, its more useful and trustworthy than Google Analytics. You can block it if you’d like.
Yes, I realize that as a data scientist who works with data not dissimilar to that being collected by these tracking scripts, I’m sort-of reducing the amount of data I get to analyze and putting myself out of a job. But its ethically essential to publish the fact that opt-out is available, for my own peace of mind and also to rebuke scare-mongers such as this ridiculous article.

Reading really large files in Python

1 Reply

There are many ways to read files in python, which is kinda un-intuitive and against the PEP-8 guidelines. However, this is the best:

with open("my_large_file.txt", "r") as f:
    for line in f:
        (code here)

with open("my_large_file.txt", "r") as f:

for line in f:

(code here)

It creates a generator which only reads one line at a time into memory. I’ve used this code to process multiple-gigabyte files with ease. Using with is useful since it automatically closes the file after the code indented below it has finished.

Often people make the mistake of using:

for line in f.read() – this loads the entire file at once and reads per character
for line in f.readlines() – this loads the entire file into a list in memory

ikigomu

A Data Science, NLP and personal blog by Matthew Ruttley

Monthly Archives: June 2013

Really simple multi-processing in Python – and then linking to MongoDB

Markov Transition Matrix Equilibrium made simple in Python

Custom DefaultDicts with Lambdas in Python for creating detailed frequency tables, or anything else

Two quick fixes to make your web browsing more anonymous

Reading really large files in Python