Je viens de publier un article, MapReduce avec parallel, cat et une redirection, sur le blogue de Datacratic.
Category Archives: Programming
64-bit Scientific Python on Windows
Getting a 64-bit installation of Python with scientific packages on our dear Windows isn’t as simple as running an apt-get or port command. There is an official 64-bit Python build available but extensions like numpy, scipy or matplotlib only have official 32-bit builds. There are commercial distributions such as Enthought that offer all the packages built in 64-bit but at around 200$ per license, this was not an option for me.
Stumbled upon the Python Extension Packages for Windows page that contains dozens of extensions compiled for Python 2.5, 2.6 and 2.7 in 32 and 64 bits. With these packages, I was able to get a working installation in no time.
My time at Sun Labs and pyaura
My internship at Sun Microsystems Labs, which has been going on for about 15 months – 9 of those full time at their campus in the Boston area – is coming to an end. During the course of those months, I’ve met a lot of very smart and fun people, I’ve worked on very challenging and stimulating problems and I’ve discovered a bunch of really good New England beers.
All my work has been centered around the Aura datastore, an open-source, scalable and distributed recommendation platform. The datastore is designed to handle millions of users and items and can generate content-based recommendations based on each item’s aura (aka tag cloud).
Last summer, under the supervision of Paul Lamere, I worked a lot more on our music recommendation web application, called the Music Explaura and designed a steerable recommendation interface. (We also have a Facebook companion app to the Explaura that was created by Jeff Alexander.)
This summer, I worked with Steve Green on many different things, including what I’d like to talk about in this post, pyaura, a Python interface to the datastore.
pyaura
The idea behind pyaura is to get the best of both world. While the datastore is very good at what it does – storing millions of items and being able to compute similarity between all of them very quickly – the Java framework surrounding it is a bit too rigid to quickly hack random research code on top of it. While my actual goal was to experiment with ways of doing automatic cleanup and clustering of social tags, I felt I was missing the flexibility I wanted and was used to getting when working on projects using Python’s interactive environment.
Without going into details, since the datastore is distributed and has many different components, it uses a technology called Jini to automatically hook them all up together. Jini takes care of automatic service discovery so you don’t have to manually specify IP adresses and so on. It also allows you to publicly export functions that remote components can call. A concrete example would be the datastore head component allowing the web server component to call it’s getSimilarity() function on two items. The computation goes on in the datastore head and then the results get shipped across the wire to the web server so it can serve its request. However, Jini only supports Java leaving us no direct way to connect to the datastore using Python.
After looking around for a bit, I stumbled upon a project called JPype, which essentially allows you to launch a JVM inside Python. This allows you to instantiate and use Java objects in a completely transparent way from within Python. Using JPype, I built two modules which together, allow very simple access to the datastore though Python.
- AuraBridge: A Java implementation of the Aura datastore interface. The bridge knows about the actual datastore because it can locate it and talk to it using Jini.
- pyaura: A set of Python helper functions (mostly automatic type conversion). pyaura instantiates an AuraBridge instance using JPype and uses it as a proxy to get data to and from the datastore.
Example
To demonstrate how things become easy when using pyaura, imagine you are running an Aura datastore and have collected a lot of artist and tag information from the web. You might be interested in quickly seeing the number of artists that have generally been tagged by the each individual tag you know about. With these few lines of code, you can get a nice histogram that answers just that questions:
import pyaura.bridge as B import pylab as P aB = B.AuraBridge() counts = [len(tag.getTaggedArtist()) for tag in aB.get_all_iterator("ARTIST_TAG")] P.hist(counts) |
The above code produces the following plot:
This is the result we expect, as this was generated with a datastore containing 100,000 artists. As less and less popular artists are added to the datastore, the effects of sparsity in social data kick in. Less popular artists are indeed tagged with less tags than popular artists, leading to the situation where very few tags were applied to more than 5000 artists.
This is a small example but it shows the simplicity of using pyaura. With very few lines of code, you can do pretty much anything with the data stored in Aura. This hopefully will make the Aura datastore more accessible and attractive to projects looking to take advantage of both its scalability and raw power as well as have the flexibility to quickly hack on top of it.
Using filemerge for mercurial diffs
A friend of mine found a script that brings up OSX’s FileMerge program instead of the text-based file comparisons you get with mercurial with doing an “hg diff”.
- download this script and make sure its location is in your PATH
- add the following to .hg/hgrc:
[extensions] hgext.extdiff = [extdiff] cmd.opendiff = fmdiff
- Now type hg opendiff <filename> (hg op is enough), instead of hg diff <filename>
SQLite3, php5 and MAMP
The version of php5 bundled with MAMP doesn’t come with sqlite3 support built-in out of the box. Here’s how to add it, considering the following setup :
- SQLite3 installed through MacPorts
- php5 used through MAMP
I used the php-sqlite3 extension. When you run phpize, make sure you are actually using the one from MAMP by calling it with its full path /Applications/MAMP/bin/php5/bin/phpize.
I then complied the extension with the following flags (both paths are the defaults for macports and MAMP) :
./configure --with-sqlite3=/opt/local/ --with-php-config=/Applications/MAMP/bin/php5/bin/php-config |
The make install didn’t copy the librarie to the right php folder :
>make install Installing shared extensions: /Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/ |
Make sure that folder is the right one. You can find your dynamic extensions folder in your php.ini file.
Ways of setting a CSS property to an element in GWT [UPDATED]
When I needed to add or modify a CSS property of an element in Google Web Toolkit (GWT), I’d always do it like this :
widget.getElement().setAttribute("style", "align:right;"); |
However, if there was already any style information for defined for that element, it would be overwritten. You could get the current attributes first and then readd them along with your new info but there is a much cleaner way :
widget.getElement().getStyle().setProperty("align", "right"); |
Important! Note that when setting a style property, your need to use camel case for the property name.
//WRONG! myWidget.getElement().getStyle().setProperty("background-color", "red"); // RIGHT myWidget.getElement().getStyle().setProperty("backgroundColor", "red"); |
GWT, state and external application-handled links
I came across the problem of having to open an external link (handled by a specific application, in this case Spotify) from a GWT application. The link does not load an actual page in the browser but only triggers Spotify to play songs from the artist specified in the link. For example, for artist Howie Day, this was already implemented in my application simply as :
new HTML('<a href="spotify:track:6g4tI7dBW9Kz4RxRjO2EC9"> <img src="play.jpg" /></a>'); |
This worked fine until I had to add a ClickListener to the HTML object that did a remote procedure call. When I would click on the link, the RPC would be made but before the answer came back from the server, the href event would be fired in the client, making GWT lose its state and causing a Status Code Exception in the browser.
An easy way to fix this would have been to open the spotify link in a new window but that wasn’t the cleanest thing to do as it would open a blank window in the user’s browser each time he would click on one of those links.
The solution I came up with was to create an invisible IFrame on my page and make the link target the IFrame, leaving my main window’s location unchanged and thus preserving GWT’s state :
new HTML('<a href="spotify:artistId" target="iframeName"> <img src="play.jpg" /></a>'); |
How does your web page look in every browser? (Updated)
Every web designer knows that making a web page come out just right in every browser can cause quite a headache, especially when combining elements like W3C standards and IE6. It’s hard to have a working copy of all the different browsers and all the different versions to test. Browsershots.org to the rescue!
Browsershots makes screenshots of your web design in different browsers. It is a free open-source online service created by Johann C. Rocholl. When you submit your web address, it will be added to the job queue. A number of distributed computers will open your website in their browser. Then they will make screenshots and upload them to the central server here.
Works great and they have 15 different browsers running on Linux, Mac OS, Windows and FreeBSD.
My blog came out perfectly on most browsers and platforms, even exotic ones like Kazehakase. Luckily for me, Microsoft was there to save the day, or else I wouldn’t have had any pictures to show.
Update : Anoter website, IE NetRenderer, allows you to get instant screenshots of your site using different version of IE.
Compile against libraries installed with MacPorts
MacPorts installs packages in a non unix-standard location. This can cause problems when trying to compile other software against these packages because they aren’t found by the configure script.
I experienced that problem when trying to compile SDL_image, which depends on libraries such as libpng. I had to configure with the following flags :
./configure CPPFLAGS="-I/opt/local/include" LDFLAGS="-L/opt/local/lib" |
which indicates in which folders the libraries installed with MacPorts were installed.