Hacking an epic NHL goal celebration with a hue light show and real-time machine learning

See media coverage of this blog post.

In Montréal this time of year, the city literally stops and everyone starts talking, thinking and dreaming about a single thing: the Stanley Cup Playoffs. Even most of those who don’t normally care the least bit about hockey transform into die hard fans of the Montréal Canadiens, or the Habs like we also call them.

Below is a Youtube clip of the epic goal celebration hack in action. In a single sentence, I trained a machine learning model to detect in real-time that a goal was just scored by the Habs based on the live audio feed of a game and to trigger a light show using Philips hues in my living room.

The rest of this post explains each step that was involved in putting this together. A full architecture diagram is available if you want to follow along.

 

The hack

The original goal (no pun intended) of this hack was to program a celebratory light show using Philips hue lights and play the Habs’ goal song when they scored a goal. Everything would be triggered using a big Griffin PowerMate USB button that would need to be pushed by whoever was the closest to it when the goal occurred.

That is already pretty cool, but can we take it one step further? Wouldn’t it be better if the celebratory sequence could be triggered automatically?

As far as I could find, there is no API or website available online that can give me reliable notifications within a second or two that a goal was scored. So how can we do it very quickly?

Imagine you watch a hockey game blindfolded, I bet you would have no problem knowing when goals are scored because a goal sounds a lot different that anything else in a game. There is of course the goal horn, if the home team scores, but also the commentator who usually yells a very intense and passionate “GOOOAAAALLLLL!!!!!”. By hooking up into the audio feed of the game and processing it in real-time using a machine learning model trained to detect when a goal occurs, we could trigger the lights and music automatically, allowing all the spectators to dance and do celebratory chest-bumps without having to worry about pushing a button.

 

Some signal processing

The first step is to take a look at what a goal sound looks like. The Habs’ website has a listing of all previous games with ~4 minutes video highlights of each game. I extracted the audio from a particular highlight and used librosa, a library for audio and music analysis, to do some simple signal processing. If you’ve never played with sounds before, you can head over to Wikipedia to read about what a spectrogram is. You can also simply think of it as taking the waveform of an audio file and creating a simple heat map over time and audio frequencies (Hz). Low-pitched sounds are at the lower end of the y-axis and high-pitched sounds are on the upper end, while the color represents the intensity of the sound.

We’re going to be using the mel power spectrogram (MPS), which is like a spectrogram with additional transformations applied on top of it.

You can use the code below to display the MPS of a sound file.

This is what the MPS of a 4 minutes highlight of a game looks like:

highligh

mel power spectogram of a 4 minutes highlight

 

Now let’s take a look at an 8 seconds clip from that highlight, specifically when a goal occurred.

goal

mel power spectrogram of a goal by the Canadiens

 

As you can see, there are very distinctive patterns when the commentator yells (the 4 big wavy lines), and when the goal horn goes off in the amphitheater (many straight lines). Being able to see the patterns with the naked eye is very encouraging in terms of being able to train a model to detect it.

There are tons of different audio features we could derive from the waveform to use as features for our classifier. However, I always try to start simple to create a working baseline and improve from there. So I decided to simply vectorize the MPS, which was created by using 2 second clips with frequencies up to 8KHz with 128 Mel bands at a sampling rate of 22.5KHz. The MPS have a shape of 128×87, which results in a feature vector of 11,136 elements when vectorized.

 

The machine learning problem

If you’re not familiar with machine learning, think of it as building algorithms that can learn from data. The type of ML task we need to do for this project is binary classification, which means making the difference between two classes of things:

  • positive class: the Canadiens scored a goal
  • negative class: the Canadiens did not score a goal

Put another way, we need to train a model that can give us the probability that the Canadiens scored a goal given the last 2 seconds of audio.

A model learns to perform a task through training, which is looking at past examples of those two classes and figuring out what are the statistical regularities in the data that allow it to separate the classes. However, it is easy for a computer to learn things by heart. The goal of machine learning is producing models that are able to generalize what they learn to data they have never seen, to new examples. What this means for us is that we’ll be using past games to train the model but what we obviously want to do are predictions for future games in real-time as they are aired on TV.

Building the dataset

As with any machine learning project, there is a time when you will feel like a monkey, and that is usually when you’re either building, importing or cleaning a dataset. For this project, this took the form of recording the audio from multiple 4 minute highlights of games and noting the time in the clip when a goal was scored by the Habs or the opposing team.

Obviously, we’ll be using the Canadiens’ goals as positive examples for our classifier, since that is what we are trying to detect.

Now what about negative examples? If you think about it, the very worst thing that could happen to this system is for it to get false positives (falsely thinking there is a goal). Imagine we are playing against the Toronto Maple Leafs and they score a goal and the light show starts. Not only did we just get scored and are bummed out, but on top of that the algorithm is trolling us about it by playing our own goal song! (This is naturally a fictitious example because the Leafs are obviously not making the playoffs once again this year) To make sure that doesn’t happen, we’ll be using all the opposing team’s goals as explicit negatives. The hope is that the model will be able to distinguish between goals for and against because the commentator is much more enthusiastic for Canadiens’ goals.

To illustrate this, compare the MSP of the Habs’ goal above with the example below of a goal against the Habs. The commentator’s scream is much shorter and the goal horn of the opponent’s team amphitheater is at very different frequencies than the one at the Bell Center. The goal horn only goes off when the home team scores so the MSP below is taken from a game not played in Montréal.

goal-against

mel power spectrogram of a goal against the Canadiens

 

In addition to the opposing team’s goals, we’ll use 50 randomly selected segments from each highlight that are far enough from an actual goal as negatives, so that the model is exposed to what the uneventful portions of a game sound like.

False negatives (missing an actual goal) are still bad, but we prefer them over false positives. We’ll talk about how we can deal with them later on.

Note that I did not do any alignment of the sound files, meaning the commentator yelling does not start at exactly the same time in every clip. The dataset ended up consisting of 10 games, with 34 goals by the Habs and 17 goals against them. The randomly selected negative clips added another 500 examples.

 

Training and picking a classifier

As I mentioned earlier, the goal was to start simple. To that effect, the first models I tried were a simple logistic regression and an SVM with an rbf kernel over the raw vectorized MPS.

I was a bit surprised that this trivial approach yielded usable results. The logistic regression got an AUC of 0.97 and an F1 score of 0.63, while the SVM got an AUC of 0.98 and an F1 score of 0.71. Those results were obtained by holding out 20% of the training data to test on.

At this point I ran a few complete game broadcasts through the system and each time the model detected a goal, I wrote out the 2 seconds corresponding sound file to disk. A bunch were false positives that corresponded to commercials. The model had never seen commercials before because they are not included in game highlights. I added those false positives to the negative examples, retrained and the problem went away.

However the AUC/F1 score were not an accurate estimation of the performance I could expect because I was not necessarily planning to use a single prediction as the trigger for the light show. Since I’m scoring many times per second, I could try decision rules that would look at the last n predictions to make a decision.

I ran a 10-fold cross-validation, holding out an entire game from the training set, and actually stepping through the held out game’s highlight as if it was the real-time audio stream of a live game. That way I could test out multi-prediction decision rules.

I tried two decision rules:

  1. average of last n predictions over the threshold t
  2. m positive votes in the last n predictions, where a YES vote requires a prediction over the threshold t

For each combination of decision rule, hyper-parameters and classifier, there were 4 metrics I was looking at:

  1. Real Canadiens goal that the model detected (true positive)
  2. Opposing team goal that the model detected (really bad false positive)
  3. No goal but the model thought there was one (false positive)
  4. Canadiens goal the model did not detect (false negative)

SVMs ended up being able to get more true positives but did a worst job on false positives. What I ended up using was a logistic regression with the second decision rule. To trigger a goal, there needs to be 5 positives votes out of the last 20 and votes are cast if the probability of a goal is over 90%. The cross-validation results for that rule were 23 Habs goals detected, 11 not detected, 2 opposing team goals falsely detected and no other false positives.

Looking at the Habs’ 2014-15 season statistics, they scored an average of 2.61 goals per game and got scored 2.24 times. This means I can loosely expect the algorithm to not detect 1 Habs goal per game (0.84 to be more precise) and to go off for a goal by the opposing team once every 4 games.

Note that the trained model only works for the specific TV station and commentator I trained on. I trained on regular season games aired on TVA Sports because they are airing the playoffs. I tried testing on a few games aired on another station and basically detected no goals at all. This means performance is likely to go down if the commentator catches a cold.

 

Philips hue light show

Now that we’re able to do a reasonable job at identifying goals, it was time to create a light show that rivals those crazy Christmas ones we’ve all seen. This has 2 components: playing the Habs’ goal song and flashing the lights to the music.

The goal song I play is not the current one in use at the Bell Center, but the one that they used in the 2000s. It is called “Le Goal Song” by the Montréal band L’Oreille Cassée. To the best of my knowledge, the song is not available for sale and is only available on Youtube.

Philips hues are smart LED multicolor lights that can be controlled using an iPhone app. The app talks to the hue bridge that is connected to your wifi network and the bridge talks to the lights over the ZigBee Light Link protocol. In my living room, I have the 3 starter-kit hue lights, a light-strip under my kitchen island and a Bloom pointing at the wall behind my TV. Hues are not specifically meant for light shows; I usually use them to create an interesting atmosphere in my living room.

I realized the lights can be controller using a REST API that runs on the bridge. Using the very effective phue library, we can interface with the hue bridge API from python. At that point, it was simply a question of programming a sequence of color and intensity calls that would roughly go along with the goal song I wanted to play.

Below is an example of using phue to make each light cycle through the colors blue, white and red 10 times.

I deployed this up as a simple REST API using bottle. This way, the celebratory light show is decoupled from the trigger. The lights can be triggered easily by calling the /goal endpoint.

 

Hooking up to the live audio stream

My classifier was trained on audio clips offline. To make this whole thing come together, the missing piece was the real-time scoring of a live audio feed.

I’m running all of this on OSX and to get the live audio into my python program, I needed two components: Soundflower and pyaudio. Soundflower acts as a virtual audio device and allows audio to be passed between applications, while pyaudio is a library that can be used to play an record audio in python.

The way things need to be configured is the system audio is first set to the Soundflower virtual audio device. At that point, no sound will be heard because nothing is being sent to the output device. In python, you can then configure pyaudio to capture audio coming into the virtual audio device, process it, and then resend it out to the normal output device. In my case, that is the HDMI output going to the TV.

As you can see from the code snippet below, you start listening to the stream by giving pyaudio a callback function that will be called each time the captured frames buffer is full. In the callback, I add the frames to a ring buffer that keeps 2 seconds worth of audio, because that is the size of the training examples I used to train the model. The callback gets called many times per second. Each time, I take the contents of the ring buffer and score it using the classifier. When a goal is detected by the model, this triggers a REST call to the /goal endpoint of the light show API.

Full architecture

goal_detector_architecture2

My TV subscription allows me to stream the hockey games on a computer in HD. I hooked up a Mac Mini to my TV and that Mac will be responsible for running all the components of the system:

  1. displaying the game on the TV
  2. sending the game’s audio feed to the Soundflower virtual audio device
  3. running the python goal detector that capture the sound from Soundflower, analyses it, calls the goal endpoint if necessary and resends the audio out to the HDMI output
  4. running the light show API that listens for calls to the goal endpoint

Since the algorithm is not perfect, I also hooked up the Griffin USB button that I mentioned at the very beginning of the post. It can be used to either start or stop the light show in case we get a false negative or false positive respectively. It was very easy to do this because a push of the button simply calls the /goal endpoint of the API that can decide what it should do with the trigger.

 

Production results and beyond

After two playoff games against the Ottawa Senators, the model successfully detected 75% of the goals (missing 1 per game) and got no false positives. This is in line with the expected performance, and the USB button was there to save the day when the detection did not work.

This was done in a relatively short amount of time and represents the simplest approach at each step. To make this work better, there are a number of things that could be done. For instance, aligning the audio files of the positive examples, trying different example length, trying more powerful classifiers like a convolutional neural net, doing simple image analysis of the video feed to try to determine on which side of the ice we are, etc.

In the mean-time, enjoy the playoffs and Go Habs Go!

 

Talks

In the media

49 thoughts on “Hacking an epic NHL goal celebration with a hue light show and real-time machine learning”

  1. Jon

    Your setup is awesome! I love it. Great work.

    As far as I could find, there is no API or website available online that can give me reliable notifications within a second or two that a goal was scored. So how can we do it very quickly?

    What APIs and/or websites have you checked ?

  2. François Maillet Post Author

    Got the two Habs goals but it did go off also for the Sens goal. But that’s in line with the performance I was expecting, which is miss 1 habs goal per game and go off for an opposing team’s goal once per 4 game.

  3. François Maillet Post Author

    Thanks. Googled a bit. Saw ESPN used to have some sort of API but it’s no longer available. Also checked IFTTT but didn’t find anything that looked suitable.

  4. Scott

    Great post. Do you have any more information on what tools/libraries you used to implement the machine learning model?

    Cheers

  5. François Maillet Post Author

    @Scott: logreg and svm in scikit learn for this initial version. Since what I did is really simple, I’m planning to improve the ML in the near future using MLDB, the ML database we’re developing at Datacratic.

  6. François Maillet Post Author

    @Alan: I don’t have an experience with a raspberri pi. But the processing and scoring took about 50% of a core of a fairly recent i7, so not sure if it would be fast enough without spending more time optimizing things.

  7. P Cote

    I understand that it will also work if the announcer goes “Scores!” or “Et le but!”

    And the algo will only get better as we make our way to the Stanley Cup!

    Next step is to detect whenever Dale Weise scores and make a special celebration.

    Go! Habs! Go!

  8. Kevin

    Awesome hack!! Any plans on open sourcing any of this project? I’d love to see how you did the machine learning/training.

  9. Mike

    This is awesome! I really like your approach to this. I’ve been working on a similar setup for the Anaheim Ducks but using a Twitter feed watching certain keywords. (We tweet “#AnaheimDucksGoooaaalll” when they score.) I’ve been recording a search of related tweets to a database and updating a webpage graph for now. The plan is to hook it up to lights, an LED matrix, and one of those red goal lights. I agree about no “realtime” way to get it from the website. Go Habs and Ducks!

  10. François Maillet Post Author

    @Kevin: It’s really a kind of unique and hacky setup so it’s not really open sourcable as is. I’ve put up most of the critical snippets in the blog post so that should get you more than started. I’m be happy to answer any questions

  11. Joe Gibson

    Wow what a cool project! Très bien!

    As a huge hockey fan (go Red Wings! We might be seeing you in the next round), this project immediately peaked my interest, and as a programmer, I was totally amazed by your project. Super cool on so many fronts! The machine learning aspect, the hardware integration. Wow! I would love to do a build for my team and see the red lights turn on when Datsyuk puts one in the net!

    Encore, très magnifique et super intéressant!

    Cheers,

    Joe Gibson

  12. Marcus Olsson

    This is just absolutely fantastic! Might steal your idea and try to build something for the upcoming IIHF World Championship (must find a way to different the commentary for the different national teams though…).

    Awesome work!

  13. furicle

    That was a lot of work – I hope you enjoy it next season. Because it’s not going to get much more work this year…..

  14. Guillaume

    Congratulations for your hack. On my side, I’ve tried the IFTTT app and the results weren’t really there… I’ve made a recipe with ESPN and Hue channels, but there was a really big delay so it was pretty useless and the effect was not so cool.

  15. MALLOW

    Bruins fan here, just gotta say this is incredible. Have always dreamed of having something like this in my living room, but I think the wife might not like it very much.

  16. François Maillet Post Author

    @Kevin: Thanks! I’ve put code snippets in the blog post with the critical parts of it. The ML of this first version is very simple. Pretty vanilla scikit learn. I’m planning on doing a follow up with a focus on improving the ML.

  17. Marc-Antoine Ross

    Wow, this is really cool. I can imagine the party. You could buy a Budweiser Red Light and reverse it… but I prefer your approach ;)

  18. Neil

    This is awesome! I’m curious why you went with processing the audio feed, rather than the video? Does Canadian hockey not have the score display in a fixed position on the screen? That would make detecting off a goal much more deterministic.

  19. Eric

    Nice work! I’m guessing that data from Habs-Sens games are probably the hardest to train with… Down here in Washington we have a lot of traitors – ugh, I mean transplants – so regular season opposition goals can be nearly as distinct as home team goals… less so in the playoffs, of course (WE ARE LOUDER!).

    How are your results if you train on more of those goals where you let more of the other team’s fans in the building?

  20. François Maillet Post Author

    @Neil: We do have the score display. However the displayed score doesn’t change within a second of a goal. Some stations have an animation and write “GOAL!!” but then you still need to figure out who scored. I’m considering improving the accuracy of the system by doing both audio and video analysis but audio seemed like an easier first step.

  21. Alex

    I’m not sure how the USB trigger works, but could you potentially record the instances of mis-classification and add them back into your training set? Then you would be improving the accuracy of the model after each game, with the model having you act as an expert witness.

  22. François Maillet Post Author

    @Alex: For sure! I did a variation on that without the button. I would play back past games in their entirety and capture every goal the system detected. Then it only a few minutes to put them in the positive or negative list. That’s the way I captured false positives in commercials and intermissions

  23. Clod

    “This is naturally a fictitious example because the Leafs are obviously not making the playoffs once again this year”
    hahahhaha :)

    Just before “Philips hue light show”, you said you tried with a different TV station.
    Can you describe a little here?
    Which one(s)? Only with habs or with a different club?
    Did you try TSN with Leafs for example?
    Can a Calgary fan uses that with CBC for example?

    Really nice stuff Francois! Are you gonna do the same for the Als? :)

  24. François Maillet Post Author

    @Clod: The training was done with games aired at TVA because they’re airing the playoffs. But I tried the system on an RDS game before the season ended. The model did not detect a single Habs goal even though there were 3 in the game if memory serves. The model locks on the TVA Sports commentator’s voice. It’s not about just triggering with there’s a loud sound; the model learned what frequency pattern represents that specific commentator yelling “Goalll!!”. Another commentator’s voice will have a different timbre, so it’ll sound different and look different to the model. So it’s a very simple voice recognition system that only detect “He scores!” or “Goal”.

    The only way to make it work for other stations would be to create a dataset for those station’s commentator. However, if you take CBC for instance, I’m pretty sure there is less of a difference between a goal for or against the Habs, especially if they’re playing against another Canadian team. The fact that the francophone stations are super biased towards the Habs probably makes it much easier for the model!

    As for the Als, let’s start by winning the cup and we’ll see about football after :) Thanks for your comment.

  25. calyth

    There are 2 things that are too bad – 1) You use a Sens vs Habs game for demo. Go Sens Go! 2) Too bad PyCon 2016 isn’t in Montréal. It would’ve been a great talk if you could demo there. They’re in Portland next year, afaik.

  26. Ethan

    Hey François, awesome set up you created! Watching that demo video never gets old.

    I’ll have to admit, I’m really intrigued in how you’re able to play the theme song through the media player to the stereo. I only know of media players receiving bluetooth. What type of media player do you use? And are you actually receiving the REST call or sending it through bluetooth?

  27. François Maillet Post Author

    @Ethan: Simpler than what you mentioned because I’m streaming the game on a Mac Mini hoked up to my home theater receiver using HDMI. So playing the goal song is done by having VLC on the mac simply play the song. I adjusted the volumes between the song and the game’s feed so they were roughly well calibrated and voilà.

  28. JF

    Nice project!

    I’ve seen no mention of the Budweiser project, similar to this but probably powered by a not-so-cool technology.

    CBC broadcasts over the air free signal and they provide live closed captions. TVA Sports must be doing that too (powered by speech recognition made in Montreal!). If the delay is too long with the closed captions, they could certainly be used to terminate FA sooner, not to mention they could provide description of the match that could help if you do some NLP and try to determine who has the puck!

    The latency is probably better with the analysis of the audio but all of this is triggered by someone manually pushing a button. You could probably augment this system if you rely on the graphics broadcasters like CBC shows. These are very expensive to create and they follow a well determined procedure when a goal is scored.

    Go Canadiens Go!

  29. Kevin Mueller

    This video inspired me to spend hundreds of dollars on hue lights and I absolutely love them. My question is: What app have you found works best for programming the light sequence and animations of the lights themselves?

    The stock hue app is rather clunky and there are too many third party apps to choose from.

    Thanks !

  30. François Maillet Post Author

    @Kevin: The lights animatino is custom python code that uses the phue library. The blog post has a code snippet illustrating it.

  31. Guus

    Thanks for this great article!

    I am trying to detect when a cup of coffee has been made by listening to the sound of the coffee machine and using machine learning. Your article looks very close to what I want to accomplish.

    After reading this article I still have a few questions I hope you can help me out with:

    How did you feed the audio signal to the ML algorithm?
    Quote: “So I decided to simply vectorize the MPS …”. What does this mean?

    Is the source code available somewhere? (I looked on your GitHub and BitBucket.) This could answer a lot of my questions.

    Your article is very interesting for my case. Unfortionalty after giving some sample code for how to generate an MPS diagram it skips over some details I really would like to know. Hopefully you can help me out :)

    Thanks in advance!

  32. Derek

    Hey- this is awesome. I have a question. I’d have to do this through the button press by the sounds of it rather than voice (for baseball, football etc)

    1) how does the computer know to cue the celebration by the usb button tap as you originally did it? Is there a software or something locally on the computer telling the media player with the goal horn plus hue to cue it up? I’m just unsure about how this part works.

    Thanks!

  33. François Maillet Post Author

    @Derek: You can get software from Griffin for the Powermate that essentially allows you to specify what to do when the button is pressed. I had the button run an AppleScript that did two things: a) run an ‘open’ shell command on the goal’s song MP3 file telling it to open it up with VLC b) do a REST call to my goal API to start the lights.

    If you only want to start a song, then maybe there’s an even simpler way in the PowerMate’s UI to do that, without using an Apple Script.

    Cheers!

  34. Kyle

    Francois I have the python program tied into my lights, they are changing the colors and flashing, how did you set it to find the starting state of the light and then return to that state after the program is ran?

  35. Kyle

    I looked at those, they helped me get it set up, but am unsure on the specific code i would need to write, I tried to retrieve the status using the b.get_light command but it is not returning a value. I can use b.set_light and set the light to on/off or color i choose. Also I am using transistiontime=0 but it is not 0, there is a delay. I want them to flash red, then to white, then alternate between white and red. during the change from red to white the turn blue then white, and from white to red the turn pink then red. Any ideas? I can send you the full code i have if you need. If you want to email me directly my email is kbostick88 at gmail. Thanks!

  36. Colin Foster

    Thanks for the idea! I’ve made a similar listening script / training algorithm because of this blog. I’ve talked around as well and for the false positives, I think you can mitigate them quite a bit by including just random noise in your negative data set. For my training, I simply made a handful of WAVs that came from numpy’s random and I haven’t seen a false positive since. Might be worth trying, in case you haven’t already.

  37. François Maillet Post Author

    @Colin: Glad to hear to made something similar inspired by this post! Was it also for hockey or some other use-case?? Anything you can share?

    Good idea for the random noise. That certainly would help in terms of expanding’s the model’s view of what’s possible and show it that vast areas of the feature space where it’s all negative. To achieve something a bit similar, since I had access to many recordings of games, I ran a bunch of them though the system at high speed and manually looked at the examples that got considered as positives, and added the bad ones to the negative examples. That took care of most of what didn’t sound similar to a goal, leaving false positives that actually sounded like goals (like goals for the wrong team).

    Cheers!

  38. Colin Foster

    Good ideas! The project was to basically log how often my wife’s cat meows. He’s really annoying and I want to quantify just how annoying he is when he wakes us up in the middle of the night :-)

    Thanks again for the idea though! It was a good learning exercise for an ML project and I really enjoyed learning more about the MFC in audio processing. I’d always wondered how services like Shazaam worked and now I think I have a pretty good understanding.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

Comment moderation is enabled. Your comment may take some time to appear.