Microsoft’s new Kinect sensor is garnering a lot of attention from the hacking community, but the technical specifics of how it works still aren’t clear. I am working to understand the technology at a fundamental level – my interest is in the optical side of Kinect. My ultimate goal is to make the sensor nearsighted, so that the depth resolution can be used to scan small objects. The first step in understanding a technology is to look at it — that’s why teardowns like this one at iFixit are so important.
Unfortunately for us humans the Kinect projector is infrared (or near-infrared) — probably somewhere between 900 and 1020nm. My guess is that it’s around 904nm, because it’s cheap to produce those lasers, but that’s just a guess; I haven’t measured anything yet. That means that by design, we can’t see the projection directly with our eyes. But we can see it with a camera. Almost all CMOS sensors in digital cameras ARE sensitive to infrared. In fact, they are so sensitive that there is a filter placed between the lens and the sensor to prevent IR from messing up your pictures. It’s called the IR cutoff filter.
The IR cutoff filter is easily identifiable. It’s a bluish piece of glass. It’s blue because the glass absorbs wavelengths that are just longer than red. This is an IR cutoff filter from an old video camera, you can clearly see how the filter just sits on top of the sensor:
Here’s the IR cutoff filter in my Sony NEX-5:
These filters are so effective in reflecting and/or absorbing IR that the reflected IR has been used to detect cameras in dark theaters! Yet another effective anti-piracy strategy… I see it’s been steadily declining since that was published in 2004.
Anyway, we need an IR sensitive camera to see what the hell is going on. I’d rather not use a cheap IR security cam — the resolution is just too low. And I’m not going to modify my NEX-5 for infrared (yet – but see Pete Ganzel’s excellent work on that!). So I’ll have to compromise and pick what is probably the most difficult option: modifying a compact Powershot camera. It’s difficult, because the filter in a compact camera is INSIDE a closed lens mechanism, and the camera was never designed to be disassembled. This tutorial will be a little incomplete, but should be enough to get any motivated person done with the same project. As all hackers know, it only takes one person to get the work done and share the results.
Start with a clean workspace. Like this one.
Disassemble the camera screw by screw. Don’t miss the hidden one inside the battery compartment.
Carefully pull the body apart:
The back will come off first. It may be blurry. Don’t let that bother you:
Next, the front:
Remove the display by removing the screw at the top left and unclipping these metal clips:
Next, carefully unclip the FFC (Flat Flexible Cables) from their mounts. Some will be pull-out, and some will have a brown plastic clip that must be flipped up to release the cable. The display will come free, but it will be embarrassed and hide from the camera.
Remove the keypad assembly, and unscrew all screws holding in the mainboard. Unfortunately, for the Powershot A540, I also had to unsolder this red and black wire which supply power to the flash. Since I hate the flash, and the camera is not dependent on it, I simply removed it for good. An IR cam has little use for a flash. Good riddance.
Under the mainboard, you can see the back of the sensor. There is a bit of hot glue on a metal plate which is supported by three screws. This is the sensor assembly; it is our target. Now, before I go much further, I need to say that the camera will be difficult to return to normal operation. The IR filter must be destroyed to get it out. This forever changes the focal characteristics of the camera unless it is replaced with flat glass (and I’m not doing that). So adjust your expectations of how perfectly the camera will focus from here on out, and all will be well.
Remove the mainboard.
Here is a better shot of the sensor assembly. The three screws are covered in hot glue. That is because next to each screw, there is a small spring which pushes up on the sensor assembly. In other words, it’s a little 3-way platform that the sensor is adjusted on. The exact position is fixed because it was calibrated at the factory for perfect focus. We are going to destroy that by removing all three screws very slowly. Slowly, because the three springs will pop out if we move too quickly.
Remove the three screws. Lift the sensor assembly free. Springs are visible.
Well, there it is. The IR cut filter, held in by two dabs of hot glue. Try to lift it with a small screwdriver or hobby knife. It will most likely break because it is very thin. Remove the little pieces. Admire their lovely blue hue.
Now comes the really hard part — you have to put the camera back together. I did so without the flash assembly in place, and I left a few screws out. My Dad taught me how to take things apart, and how to re-assemble them. He always called these “extra” screws “shipping screws”. Here are the shipping screws left out of this camera:
Alright! I’ve completed the very first tool in my Kinect hacking suite, a camera to look at the Kinect’s pattern. But the question is: Was all this worth it? Does this really tell us anything? See for yourself:
Before the infrared modification:
After the infrared modification:
Alright, looking good! Let’s make a movie:
This is kickass! Here’s what the sensor output looks like up close — this board is about .5M away from the camera (click for full resolution, slightly misfocused – this image is public domain – I make no claim to copy rights):
It’s interesting in a few different ways. One can easily see that the Kinect IR speckle-field is a 3×3 matrix of random dots. One can also see that they are differentiated by intensity and have a centered registration dot. So it is not purely random speckle.
IR cameras are good for more than Kinect hacking. Here are a few outdoor shots showing some IR effects.
Stay tuned as I build up a proper toolset for optical analysis and decipher just how these guys are using speckle to estimate depth.