This summer I had the pleasure of supervising Alex Key, a Bachelors student visiting from the University of Münster. His project involved looking at how we might auto-generate videos of cooking in the kitchen.

The idea is that perhaps I want to be able to capture the record of my making a certain dish — maybe for my own future reference, maybe to share with friends, maybe to show off online. If you have a few cameras positioned to look down onto the worktop, you can get a lot of data — but how can you make cinematographic decisions such as which camera to use, and when to cut shots?

The first question isn’t too hard, as there are plenty of ways to find out which video stream has the most movement at a given point in time (if cameras 1 and 2 show static images, but camera 3 has movement, that’s where the action is!). But what about cuts? How can we say things like “Clare’s been chopping carrots for 15 seconds now, any more and the viewers will fall asleep — cut to the next bit”?

Of course, the Ambient Kitchen at Newcastle uses those lovely WAX sensors that pick up acceleration data — being able to tell when utensils are picked up and put down is very helpful on beginning to make those decisions, and Alex used this really well.

Three camera views plus some WAX data

There’s also a great piece of extra work: what if I, as the cook, want some amount of control over the film editing? Maybe I want to be able to indicate to the kitchen “hey, that was a cool thing I just did, make sure you include the last 20 seconds in the final cut”. Well, Alex investigated a couple of different methods for indicating that to the kitchen using the instrumented utensils (moving them in certain ways) — so you don’t have to interrupt your cooking to go press a button on a screen or anything like that.

This was a fun project to supervise, but what I really like is that it once again showcases the kitchen as a platform, which can be used in any number of domains — from supporting people living with dementia, to teaching foreign languages, to sharing cooking videos.