Tag Archives: Programming

MapReduce avec parallel, cat et une redirection

Efficient log processing

I’ve recently learned a couple of neat tricks to process large amounts of text files more efficiently from my new co-worker @nicolaskruchten. Our use-case is efficiently going through tens of gigabytes of logs to extract specific lines and do some operation on them. Here are a couple of things we’ve done to speed things up.

Keep everything gziped

Often, the bottleneck will be IO. This is especially true on modern servers that have a lot of cores and ram. By keeping the files we want to process gziped, we can use zcat to directly read the compressed files and pipe the output to whichever script we need. This reduces the amount of data that needs to be read from the disk. For example:

zcat log.gz | grep pattern

If you’re piping into a Python script, you can easily loop over lines coming from the standard input by using the fileinput module, like this:

import fileinput
for line in fileinput.input():
    process(line)

**Use parallel to use all available cores**

GNU parallel is the coolest utility I’ve discovered recently. It allows you to execute a script that needs to act on a list of files in parallel. For example, suppose we have a list of 4 log files (exim_reject_1.gz, exim_reject_2.gz, etc) and that we need to extract the lines that contain gmail.com. We could run a grep on each of those files sequentially but if our machine has 4 cores, why not run all the greps at once? It can be done like this using parallel:

parallel -j4 "zcat {} | grep gmail.com" ::: exim_reject*.gz

Breaking down the previous command, we tell parallel to run, using 4 cores, the command zcat {} | grep gmail.com, where {} will be substituted with each of the files matching the selector exim_reject*.gz. Each resulting command from the substitutions of {} will be run in parallel.

What’s great about it is that you can also collect all the results from the parallel executions and pipe them into another command. We could for example decide to keep the resulting lines in a new file like this:

parallel -j4 "zcat {} | grep gmail.com" ::: exim_reject*.gz | gzip > results.txt.gz

Use a ramdisk

If you’ll be doing a lot of reading and writing to the disk on the same files and have lots of ram, you should consider using a ramdisk. Doing so will undoubtedly save you lots of IO time. On Linux, it is very easy to do. The following command would create an 8GB ramdisk:

sudo mount -t tmpfs -o size=8G,nr_inodes=1k,mode=777 tmpfs /media/ramdisk

In the end…

By using all the tricks above, we were able to considerably improve the overall runtime or our scripts. Well worth the time it took to refactor our initial naive pipeline.

64-bit Scientific Python on Windows

Boston Music Hackday

1 Reply

I was thrilled to attend the Boston Music Hackday this week-end. A lot of people hacked up some pretty cool projects, many of us coding until the very early morning Sunday (aka 4am), only to get back up a few hours later (aka 8am) to keep at it until the dreaded 15h45 deadline, when we all had to submit our demos. The organisers did a wonderful job and the event was a success at every level.

The hack I did was called the PartyLister. The goal was mainly trying to come up with a way to generate steerable playlists that would also be personalized for a group of people (ie.: taking into account each of their musical taste and making sure everyone gets a song he likes once in a while). Given the very limited amount of time available to hack this up, I had to keep things simple and so I decided to use only social tags to do all the similarity computation. I excepted the quality of the playlists would suffer but the goal was really to develop a way to include multiple listeners in the track selection process. The algorithm should then be used in conjunction with something like the playlist generation model I presented at this year’s ISMIR.

My hack: PartyLister

Imagine you’re hosting a party and using the PartyLister as DJ for the night. Each of your guests will need to supply the software with his last.fm username and we’ll be good to go.

We go out and fetch from the last.fm API the social tags associated with the artists (and their top tracks) that our listeners know about. We also use the EchoNest API to get similar artists so we can present new artists to our listeners. From a user’s top artists, we can create a tag cloud that represents the user’s general musical taste (UMT). We’re also allowing each user to specify a set of tags that represent their current musical taste using a steerable tag cloud.

Suppose you have 3 guests at your party, where two like pop and the other likes metal. By doing a naive combination of the users’ musical taste, we’ll probably end up playing pop music, leaving our metalhead friend bored. To solve this, I added a user weight term which is determined by looking at the last 5 songs that played and computing the average similarity between the user’s musical taste and those songs. If we’re only playing pop songs, the metalhead will have a very low similarity between his taste and what played and so we’ll increase his weight and lower the pop lovers’ weights. When we pick the next song, this weighting scheme will allow the metalhead’s taste to count more than the pop lovers’, even if there are more of them. This will make us play a more metal-like track. After a while the weights will equal out and we’ll start playing pop music again.

For sparseness reasons, I operated on artists instead of tracks. A simplified version how I weighted each candidate artist is below. Lambda is simply a knob to determine how much the users’ musical taste will count, cd() represents the cosine distance and UMT represents a combination of the user’s general musical taste and his steerable cloud.

$\text{cand}_i = cd(\text{seed},\text{cand}_i) + \frac{\lambda}{\sum_{\text{users}} [\text{user weight}]}\sum_\text{users} [\text{user weight}] * cd(\text{cand}_i,\text{UMT})$

The following plot represents a running average of the cosine distance (dissimilarity) between users’ musical taste and the last 5 songs that played. It represents a 160 songs playlist with 3 listeners in the system.

dave_paul_frank_160

As you can see, as a user’s running average increases, his weight is also increased so that we start playing more songs that fit his taste. His average then decreases as the other users’ weights go up forcing a return to music that fits their taste a little more. The plot shows that the system seems to be doing what we want, that is taking into account the musical taste of multiple users and playing music that each person will like once in a while. Integrated in a real playlist generation model, I believe this could produce interesting results.

I also played with a discovery setting, where users could specify if they wanted to discover new songs or stick to what they know. This was achieved by adding a bonus or penalizing each candidate’s score, based on the discovery setting (float between 0 and 1) and the proportion of users who knew (had already listened to) the artist in question.

PartyLister was not a very visually or sonically attractive hack like some of the others but I still managed to win a price based on popular vote. Thanks to all the great sponsors, there were a lot of prizes and so lots of winners.

Below is the Université de Montréal delegation, Mike Mandel (who also won a price for his Bowie S-S-S-Similarities) and myself, with our bounty.

P1020732

I really hope to attend another hackday soon as it was all a lot of fun. Time to go get some sleep now.

Using filemerge for mercurial diffs

SQLite3, php5 and MAMP

1 Reply

The version of php5 bundled with MAMP doesn’t come with sqlite3 support built-in out of the box. Here’s how to add it, considering the following setup :

SQLite3 installed through MacPorts
php5 used through MAMP

I used the php-sqlite3 extension. When you run phpize, make sure you are actually using the one from MAMP by calling it with its full path /Applications/MAMP/bin/php5/bin/phpize.

I then complied the extension with the following flags (both paths are the defaults for macports and MAMP) :

./configure --with-sqlite3=/opt/local/ --with-php-config=/Applications/MAMP/bin/php5/bin/php-config

The make install didn’t copy the librarie to the right php folder :

&gt;make install
Installing shared extensions:     /Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/

Make sure that folder is the right one. You can find your dynamic extensions folder in your php.ini file.

Ways of setting a CSS property to an element in GWT [UPDATED]

13 Replies

When I needed to add or modify a CSS property of an element in Google Web Toolkit (GWT), I’d always do it like this :

widget.getElement().setAttribute("style", "align:right;");

However, if there was already any style information for defined for that element, it would be overwritten. You could get the current attributes first and then readd them along with your new info but there is a much cleaner way :

widget.getElement().getStyle().setProperty("align", "right");

Important! Note that when setting a style property, your need to use camel case for the property name.

//WRONG!
myWidget.getElement().getStyle().setProperty("background-color", "red");
// RIGHT
myWidget.getElement().getStyle().setProperty("backgroundColor", "red");