This weekend hack is a small Kinect application - Kibloc is a physical object counter/tracker using Kinect.
Update As Of Feb 1 2012: This article is explaining Version 1.0 Beta 2 APIs. How ever, now the Official 1.0 Kinect SDK For Windows is available, and there are changes from the Beta 2 API.
Here is the video that demonstrates real time, distance based blob tracking.
In the video, the image drawing for the blob highlight is a bit slow, this is because I'm drawing it on top of the color image as discussed below. For actual games/calculations, you don't need the drawing on top of the color image/bitmap.
So, let us develop this. The steps involved are pretty simple.
Kinect API allows you to work with the Kinect camera's image/video stream, almost similar to a web cam. Apart from the RGB Video camera, Kinect has got a depth sensor as well., which gives you information about the distance of an object in front of the device - and you can leverage the Video data and Depth data together to detect objects using Kinect. Kinect also exposes built in support for Skeletal tracking. The classes for working with Video, Depth and Skeletal Tracking resides in the namespace Microsoft.Research.Kinect.Nui. You can access the Kinects connected to the device via the Runtime.Kinects enumeration, and can access the Depth, Video and Skeletal tracking data by handling the DepthFrameReady, VideoFrameReady and SkeletalFrameReady events of a connected Kinect object. In the below example you could see that we are accessing the Depth data from Kinect.
Additionally, you can also use the Audio features of Kinect, leveraging classes available under the namespace Microsoft.Research.Kinect.Audio.
That is it for an interesting, and I suggest you should go through the SDK Quick starts and samples here at Coding 4 Fun if you need to learn more about how to work with the SDK in detail. The API is intuitive and super easy, but I suggest you may go through these videos to kick start.
The first step is to initialize Kinect, so here is the self explanatory code.
Basically, you may see that we are initializing the Kinect run time by passing the flags UseColor and UseDepth, so that we can consume the DepthStream and VideoStream later. The event handlers are attached to DepthFrameReady and VideoFrameReady events as shown above.
Kinect depth data is just an array of bytes, where each pixel is represented with two bytes (16 bits). Each pixel in the depth data provides distance of a pixel from the sensor, instead of color information as in a normal bitmap pixel. You can either specify Kinect to provide you the raw depth data (ImageType.Depth) or to provide you depth data along with player index (ImageType.DepthAndPlayerIndex) when you initialize the DepthStream (see the above code). You can read this iProgrammer article if you are interested to learn more about the Depth data, in detail.
Basically, what we do in the below method is to walk through the PlanarImage (raw byte array) for Depth data, and create a bitmap with 4 bytes per pixel (Red, Blue, Green and Alpha) – where Red, Blue and Green values will be the same value to form a gray scale image. The value is any number between 0 to 255, based on the distance of a pixel.
// will represent a different color channel per pixel in the final iamge.
// a pixel in the final image, we set the index to be twice of the depth index.
Once the depth image is created, we pass it over to the Blob detector to detect the blobs and high light the same.
For blob detection, I'm using the Excellent AForge library. The BlobCounter class takes the above image, and calculate the edges/rectangles of the blobs in the image. Here is an expansion of our earlier DepthFrameReady and ImageFrameReady events, you can see that we are creating a sliced depth image based on the slider values, and creating a depth frame image bitmap and a color frame image bitmap, and passes both of them to the blob detector. We are also specifying what type of high lighting to be used by the detector, based on the combo box the user has selected.
The following ProcessImage method in our BlobDetector class basically uses the AForge library to do the detection based on depth image.
After detection, we are drawing the high lights back to the color image which results in the image above. This code is mostly based on the AForge blob detection sample.
In this post, we explored how to create a blob detector and object counter using Kinect. As of now, Kinect's built in tracking feature is limited just to Skeletal tracking, but you can use libraries like AForge.net and openCV to leverage Kinect for more out of the box scenarios. Thanks to my kiddo for lending me her dolls for the demo . Also, follow me in twitter. And here are few more posts on a similar taste.