Hi all,
Those (two) of my regular readers will know I frequently curse my fool-of-a-took laptop. With this in mind I’ve been meaning to write a tutorial on working with Amazon’s web services for when you don’t have the hardware you need. This isn’t that post but I will assume you’ve already got a cluster to hand with all the bits and bobs you need installed.
If you don’t, don’t worry about it (for now) – I’ll do a post on how to get one of those set up at a later date. For now I’d like to talk about collaborative filtering. This is one of the most prevalent techniques for generating recommendations and with good reason – it’s pretty good. It’s quite a common sense method:
I know that person A likes items 1, 2 and 3 but not item 4.
I know that person B likes item 4 and 3 but not 1 or 2.
I know that person C likes item 1 but not 2, 3 or 4.
I know that person D doesn’t like item 4 – which of the items 1, 2 or 3 should I recommend them?
I’m sure most people reading this will say we should recommend them item 1, then probably item 2 then 3? Certainly item 1 with highest preference. Now if you deconstruct your thought process in coming up with that recommendation I’d wager (not a lot) that it’d be akin to the collaborative filtering algorithm.
Based on the behaviour of people A, B and C we can start to get an idea about which items are rated favourably by the same people versus those that aren’t. User-based collaborative filtering is about finding similar users to person D and looking at what they liked. Item-based collaborative filtering is about finding similar items to the ones person D has rated positively and recommending those.
There are lots of nice articles on collaborative filtering – more often than not when it comes to machine learning I end up recommending Andrew Ng material and this is no exception – check out his machine learning course on Coursera. There’s a whole lecture on collaborative filtering and it’s boss. If my explanation wasn’t enough (if you’re serious about this sort of thing, it shouldn’t be) check out his lectures on the subject and read around on the mathematical details of it all. Make sure you polish up on your linear algebra – you’ll have very large sparse matrices heading your way!
Also, please note that this whole post is just a re-imagining of this post by Amazon: Recommendations using EMR – I liked it and thought I’d give it a go myself with a different data set and technologies I was more familiar with. Now on with the show…
The Data Set
I’m going to be using a publicly available data set containing user ratings of books gathered on Amazon – dataset available here: Book Data Set
There are three files in there – one with all the details of the books, one with all the details of the users and finally one with all the user ratings. Don’t worry about downloading them yet if you’re following this as a tutorial/example – we’ll stick that bit in a script later on.
The Cluster Set-Up
Given that I’m on the free tier and I don’t use AWS at work (we’ve got a nice cluster that handles all our data just fine thank you) I’m a bit ‘over-compute’ happy at the moment and so I’m going to use a 5 node cluster (1 master node and 4 slaves). This isn’t really necessary but then nor is this as a piece of work so I’m not that bothered by practicality at this point. The benefit of using Amazon’s Elastic Mapreduce is that the nodes all come with Hadoop (& Hive/Pig/Mahout/whatever) installed and set up so you don’t have to bother with any of that.
Building a Service
I want to demonstrate how easy it is to turn this sort of thing into a recommendations service (instead of a one-off bit of analysis). Now I’m going to stress that I’m setting this up as an offline batch machine learning method and that there are a few steps I’d change if I were actually setting this up as a real, ‘production’, service.
I’m going to use a Redis database because I’ve got a reasonable amount of memory on these boxes (Redis is an in-memory key-value store), because Redis is really really quick and because I’m used to using Redis with Python. Redis will store the output of my collaborative filtering and I’ll stick a lightweight web framework (Flask) on top of it. This will allow a client to retrieve recommendations for a given user as well as delete them or overwrite them. I’ve decided to add the latter two functions because we don’t live in a perfect world.
I’ll also write a script that’ll update the books, users and ratings files and stick it on a cronjob to run every day. That way, by copying this method you’ll end up with a reasonably scalable, reasonably fast and reasonably up-to-date recommendation service. It’s not the best one out there (heck, it might be the worst) but it goes to show how simple these things are to set up given all the tools that are already written…
The Code
Firstly, we’ll create the cluster using the Elastic Mapreduce CLI:
./elastic-mapreduce --create --alive --name recommendation-service --num-instances 5 --master-instance-type m1.large --slave-instance-type m2.xlarge --ami-version 3.1 --ssh
The --ssh means that once the cluster has been created and all the necessary bits and bobs installed we ssh into the master node
Extracting the data set
wget http://www2.informatik.uni-freiburg.de/~cziegler/BX/BX-CSV-Dump.zip
unzip BX-CSV-Dump.zip
sed '1d;' BX-Book-Ratings.csv | tr ';' ',' | tr -d '"' > Book-Ratings.csv}
Unfortunately for us, we’ve got a bit of work to do – the ‘out of the box’ Mahout recommendation algorithm requires user and book ids to be integers and ISBNs are not. To get around this, we use this nifty little script:
#!/usr/bin/python
import sys
import re
isbn_dictionary = {}
counter = 0
with open('Book-Ratings.csv', 'rb') as f:
with open('New-Book-Ratings.csv', 'w') as g:
with open('Book-Mappings.csv', 'w') as h:
for line in f:
try:
user, book, rating = line.split(',')
except:
continue
try:
index = isbn_dictionary[book]
except:
isbn_dictionary[book] = counter
index = counter
h.write("%s,%dn" % (book, counter))
counter += 1
g.write("%d,%d,%dn" % (int(user), index, int(rating)))
Now we’ve got data in the format our Mahout recommender is expecting let’s put it in HDFS and run our algorithm
hadoop fs -put New-Book-Ratings.csv /New-Book-Ratings.csv
mahout recommenditembased --input /New-Book-Ratings.csv --output recommendations --numRecommendations 10 --similarityClassname SIMILARITY_COSINE
Now we play the waiting game…
Actually, while that’s running, open up another SSH session to that terminal and run these commands
sudo easy_install redis flask
wget http://download.redis.io/releases/redis-2.8.13.tar.gz
tar xzf redis-2.8.13.tar.gz
cd redis-2.8.13
make
./src/redis-server &
cd
Now Redis is installed and running and we’ve got the necessary Python libraries we can create the following server…
import redis
import os
isbn_dictionary = {}
with open('Book-Mappings.csv', 'rb') as f:
for line in f:
isbn, counter = line.split(',')
isbn_dictionary[int(counter)] = isbn
r = redis.StrictRedis(host='localhost', port=6379, db=0)
p = os.popen('hadoop fs -cat recommendations/part*')
for recommendation in p:
user, recommendations = recommendation.split('t')
recommendations = [entry.split(':')[0] for entry in recommendations.replace('[', '').replace(']','').split(',')]
recommendations = [isbn_dictionary[int(entry)] for entry in recommendations]
r.set(int(user), recommendations)
Now we’ve populated our database with all the recommendations we generated with our Mahout job. Now we’ll set something up (called webserver.py) to return these items to a client (as well as allow modifications to be made).
import redis
from flask import Flask
r = redis.StrictRedis(host='localhost', port=6379, db=0)
app = Flask(__name__)
@app.route('/user/<user_id>', methods = ['GET', 'POST', 'PUT', 'DELETE'])
def restful_api(user_id):
if request.method == 'GET':
recommendations = r.get(int(user_id))
return recommendations if recommendations else "User %d does not exist" % int(user_id)
elif request.method == 'POST':
if request.headers['Content-Type'] == 'application/json':
r.set(int(user_id), request.json)
return "Successfully set recommendations for user %d" % int(user_id)
else:
return False
elif request.method == 'DELETE':
if r.exists(int(user_id)):
r.delete(int(user_id))
return "Successfully deleted recommendations for user %d" % int(user_id)
else:
return "User %d does not exist" % int(user_id)
elif request.method == 'PUT':
if request.headers['Content-Type'] == 'application/json':
r.set(int(user_id), request.json)
return "Successfully set recommendations for user %d" % int(user_id)
else:
return "Require JSON put"
else:
return "No idea what you've done here, but don't do it again"
if __name__ == '__main__':
app.run()
Now we’ve built our client-facing recommendations service let’s ‘deploy’ it and give it a test
python webserver.py &
curl -X GET localhost:5000/user/277965
curl -X DELETE localhost:5000/user/277965
curl -X GET localhost:5000/user/277965
Summary
We’ve not done anything that tricky but in a fairly short time we’ve managed to build something that’d be at least part way useful to a small business somewhere looking for recommendations for their users. It’s not a great leap from this and a genuine production ready system – you don’t need 5 Hadoop nodes running all the time for a 20 minute Mahout job and I’m uncomfortable using Flask and local files for a serious system. As such, you’d likely put together a couple of nodes, each with their own copy of the recommendations set in memory. One of them would create a cluster, get any new data a generate the data set as often as you saw fit – communicating it to the other node. Finally, you’d add a few more actions to the server and make sure it could be called by whichever clients need it (and block everybody else).
As I say, nothing earth-shattering but a solid base on which to build a recommendations system.
I am become death.
