Tutorials

RGBDToolkit is hardware and software which combines the images from a standard video camera with data captured from a depth sensor like the Microsoft Kinect or the Asus Xtion Pro Live.

The toolkit mount firmly connects the cameras together to fix the relationship between their lenses. The capture software is then used to calibrate the lenses, using a mathematical model of their field of views and relationship to one another. Recording the data streams to disk, the final step of visualization allows you to render the data in a 3D application and export the results.

BUILD

These are instructions for making a durable mount to attach an Xbox Kinect to the base of a HDSLR or conceivably any camera that has a tripod mount. The mount is designed with the following constraints; that the relationship between the Kinect and the camera stays fixed and isn't susceptible to bumping or slipping, the lenses are as close as possible without obscuring one-another, the whole system can easily be mounted on a tripod.

This is by no means the only way to do this. There are aspects of this system that could use improvement: most notably the mechanism for adjusting the angle of the Kinect in the mount. We would be thrilled to see your modifications, additions or complete redesigns of this mount.

Tools

  • Drill press or household drill
  • Table, hand or miter saw - as long as you can safely make straight cuts in aluminum any of these should work
  • Misc. screw drivers & wrenches
  • Lubricant (see instructions)
  • Clamps
  • 1/4"-20 tap - If you are using a drill press I recommend getting the all-in-one screw+tap sets.
  • 10-24 tap
  • 1" hole saw
Camera Mount - Left

Materials

Qty Item Notes
1 Camera Quick Release Plate Adapter Set This one differs from the Manfrotto vserion in that it is less expensive but not threaded. It has a 1/4" counter-sunk hole in the top instead of the threaded 3/8". If you would like to use your existing Manfrotto quick release, see note below.
4 1/4"-20 x 3/8" screws
1 1/4"-20 x 1" counter sunk screw
5 1/4" lock washers We recommend using these to prevent overtightening with the soft aluminum.
4 10-24 1/2" thumb screws Can be substituted with regular screws
1 15" of 2" angle aluminum We recommend getting plenty extra in case you make mistakes.
1 15" of 2" angle aluminum We recommend getting plenty extra in case you make mistakes.
1 1/4"-20 nut
Coarse sandpaper
Self-adhesive silicone bumpers
2 Blank Silicone Wristbands

Instructions

Cut the angle aluminum into three pieces - 2", 4.5" & 7" long.

2-Inch Piece:

Drill 2 x 1/4" holes 1" apart on a line that is 1" from the apex of the angle aluminum.

Drill & tap a 1/4"-20 sized hole in the center of the opposite side.

4.5-Inch Piece:

Place the 2" piece on the inside of the 4.5" piece so that they form a “C". Align the smaller piece in the center of the larger and mark the holes to match those that you've already drilled and score the edges on the larger piece.

Tap those two holes with the 1/4"-20 sized tap and check the fit with screws.

Drill a 1/4" hole that is mid-way between the mark you just scored and the outside edge and 1/4" from the inside corner of the angle aluminum.

Place the quick-release plate you're choosing to use in the center of the opposite side, lining up the 'front' (opposite the release lever) with the apex edge of the aluminum and mark the location of the hole and drill a hole to match.

7-Inch Piece:

Place the 4.5" piece against the 7" piece in the way that they'll eventually be put together (see diagram.)

Place it so that the top surface measures 0.85" from the top edge of the 7" piece and make a mark through the two holes. It is important that it is level, so measure both sides carefully.

Tap those marks with the 1/4"-20 bit and make sure the two parts fit together.

Cutting a hole for the Kinect 'foot':

Next step is making space for the base of the kinect. This could be done in a few ways. I used a 1" hole saw, then a band saw to cut them out. When you do this part make sure that you clamp down the aluminum to your work surface and lubricate your bit well. An alternative method could be to make two cuts in about 1.5" from the edge outside edge, use a vise-grips to bend that portion out/down, then cut it off and file off the excess.

RGBD ToolkitAdjustment mechanism:

The last step is to make adjustable 'risers' to change the angle of the Kinect. I have done this by tapping holes in the the 'feet' of the mount with the 10-24 bit and cutting 1"-wide strips from the angle aluminum as inserts. See photos.

Finishing touches:

Add the silicone bumpers on the inserts and the back wall of the mount. Then, in order to mount the kinect, double-wrap it with the silicone wristbands. There are various ways you can do this, but make sure that the kinect is firmly pressed against the bumpers and will not slide or twist. Another solution is to semi-permanently attach the mount to the kinect with some industrial strength Velcro. This shouldn't present a problem if you - for some reason - decide to use your Kinect to play video games down the line.

Notes

If you already have a Manfrotto quick release plate and adapter on an existing tripod that you'd like to use instead of the kind that I recommend above, you'll have to drill a 3/8" sized hole in the aluminum base and buy a 3/8"x1/2" hex cap screw.

When screwing, tapping or sawing aluminum make sure to lubricate the area first. This will prevent it from galling or tearing the aluminum and will keep the aluminum pieces from fouling your tools. You can use WD-40 or machine oil - even olive oil would work fine.

We owe a debt of gratitude to Thomas Haggerty for working with us to design the first generation of this mount, upon which all following mounts have been modeled.

CALIBRATE

Calibration process "locks" the relationship between Kinect and the DSLR. This process matches the optical characteristics of sensors and lenses of the two cameras and determines their relative positions to one another. Once calibrated, you can use the same pair again and again without recalibrating, but the Kinect and SLR must not move in relationship to one another. For that reason we built the custom mounting rig for both cameras. You can order one from us or create your own from the instructions. You can also just put cameras on the table/shelf/floor but if they move even the slightest amount you'll need to redo the calibration.

Calibration process requires fair amount of IR light. The easier and cheapest way to get good IR light is to calibrate outside in the sun. If that's not an option you can use external led IR lights or bright film lights. LED panels and incandescent bulbs wont' work because they contain very little IR light.

We recommend getting IR lights like this one

To create a calibration we take pairs of images from each camera looking at a checkerboard. Shooting is done so that we capture pairs of images using Kinect and DSLR. It is important to remember that we need to shoot in video mode on DSLR and not in photo mode. If you forget this calibration will not work. Zoom on the DSLR should be at the widest setting, and it is not bad idea to put little piece of tape on zoom ring so that you make sure it will not move. If you accidentally move the ring you'll need to redo the calibration process.

Print and mount this checkerboard (https://github.com/downloads/obviousjim/RGBDToolkit/chessboard_a4.pdf) to hard wood or plexi. It is very important to use this particular checkerboard image as the algorithm is searching for exactly that number of corners. It also needs to be completely flat and rigid. During calibration place mount the board to a sturdy stand. It can not be hand held by hand.

Typically it takes between 13 and 19 pairs of images to get good calibration. We shoot 3 depths: the close one (30-40cm from the lens) takes only one exposure (checkerboard almost fills the frame), middle (80-90cm from the lens) and far (150-200cm from the lens). At each distance other than close, pan the tripod ones take 9 exposures each: Top left, top right, top left, middle right, centered ... all the way to bottom right.

You can move the checkerboard or the camera. It doesn't matter which one you move as long as you don't mess the relationship between Kinect and the camera (this is VERY important, so we will repeat this over and over).

Here is a step by step process to create the calibration

Preparation

  • Attach the Kinect to the SLR using the quick release bracket and mount and place the pair on a video tripod with a fluid head.
  • Cover Kinect IR laser emitter
  • Plug your Kinect into your computer.
  • Launch RGBDCapture program.
  • Click on the big top left button to set the working directory. All your calibration files will go into a subfolder called calibration/ and the footage into a folder called takes/ .
  • Make sure you've highlighted the CALIBRATION tab on the top left. You should see the raw IR image (as opposed to the depth map) in the preview window.
  • Switch on the DSLR and set it to live view video mode.
  • position the checkerboard in the center of the view of the SLR camera. Using the small screws on the bottom of the mount, adjust the tilt of the Kinect until the checkerboard also appears in the center of the Kinect view. Ideally the SLR's frame will cover more than the Kinect's view, thus providing color data for the entirety of the depth image.
  • Under proper lightning, you should see the array of evenly spaced red dots on the checkerboard. If any of them are off the grid or wiggling due to video noise it indicates the light is either too strong or too weak or the checkerboard is too far.
  • FOCUS your DSLR on the checkerboard, again being careful not to touch the zoom.
  • Begin capturing pairs
Calibration Process

Pair Capturing

At each checkerboard position record one very short video and capture one calibration image from the Kinect in RGBDCapture using the spacebar. The calibration image from kinect appears on the bottom of the screen. Make sure to keep the SLR clips short as we will just be extracting single frames from them. Wait a second after pressing record to hit space bar on the computer, as inevitably shake the rig a little bit when you press record on the DSLR. You may also consider using the Canon software utility to trigger recording or a physical remote.

Repeat this process for every position of the checkerboard. As you capture images, pay attention to the reprojection error above the each calibration image inside of RGBDCapture. The value should be between 0.0 and 0.3 - the lower the value the better the calibration. If you get a high number in there (most likely due to bad lighting or the board being too far away), adjust the board slightly position and reshoot the same position. It's super important that every time you capture a calibration image on the software you also shoot a short clip on the SLR, this way the number of takes will be the same and it will be much easier to pair them in the next step. It's also a good idea to match the time on a DSLR to the time on a computer as accurate as possible. That way pairs will have the same creation time.

REMEMBER TO REFOCUS when you change the position of the checkerboard.

RGB Calibration Frame extraction

The next step is to extract single PNG images from every large video file for us in the calibration.

  • Click on the "Load RGB Calibration Movies" button
  • When prompted select the folder where you placed DSLR footage containing the calibration clips.
  • Wait while the script runs to extract a PNG from each calibration clip. The interface will update with a row of images below the kinect calibration images
  • Make sure that you have the same number of images in both rows. If not, go into the _calibration folder in the MediabBin working directory and match them up. The images are stored in depthCalibration and colorCalibration respectively.
  • After the pairs are matched click "Regenerate Calibration" and it will calculate all the error parameters.
  • You may get images that go red, indicating the checkerboard was not found in that image. Select these pairs and delete them by clicking on the pair (it will highlight in red outline) and press delete.
  • Check for pairs with high error values (anything greater than 0.3). Select those pairs and erase them as well.
  • After deleting all the missed checkerboards and images with high error, click "Regenerate Calibration" again and then press "Save Calibration". the _calibration folder is then populated with a _matrices
  • When all the high error frames have been removed, click "Save Calibration" which will save the series of .yml files into your selected working directory. it will also copy the images that were used for the calibration there too so there too which you should back up.
RGBD Calibration Process

Congratulations, you're finished with the calibration process. It's time to shoot!

CAPTURE

Capturing footage with RGBD
  • in the RGBDCapture app switch to the middle tab: Record.
  • Make sure you've remove the cover from the IR projector
  • To begin capturing footage press space bar on the computer and start the recording on DSLR.
  • Since the recordings will never begin at the same time we need to create some sort of distinct gesture to synchronize them later. We tend to have our subject or someone from the crew clap their hands firmly and quickly.
  • Make magic moments happen in front of the rig. Space, color, and time are your playground.
  • When you finish stop both recordings (press space bar on computer). It's good to get a finishing synchronization clap as well to check against.
  • NOTE: if you see the RECORD bar underneath the application flashing or filling red it indicates there are frames in the buffer that are not saving. This can result in dropped frames or lost footage. To avoid this write to a fast hard drive and make sure all other applications are closed, that you have plenty of hard drive space. If you are writing to an external hard drive we recommend FireWire over USB
  • When not filming the application is compressing the depth frames in the background to preserve disk space. This is indicated by the gray bar below each Take button. Keep the app open until all the clips have been converted.

MEDIA BIN PREPARATION

After filming we need to add the RGB recordings to the MediaBin.

The folder structure for the media bin looks like this:

MediaBin/
    _calibration/
        _matrices/
            6x .yml files generated from the calibration step
    TakeOne/
        depth/
            frame_000_millis_00000.png
            ....
        color/
            MyClip.mov //the movie clip that corresponds to this
            MySLRCLip_Small.mov //optional downresed version for offline
    TakeTwo/
        ...
  • copy each recorded clip into the color/ folder for that take.
  • You can have both on online and offline version of our video so visualization is performant. Download mpeg stream clip (or use something like Quicktime Pro or Compressor to batch conversion program). Convert the clip to to a 640x360 (assuming you shot in 16:9 aspect ratio) MJPEG @50% speed and remember to include sound. This is the clip that RGBDVisualize will load for preview, which it will then be swapped out automatically for rendering.

VISUALIZE

Each Scene can have any number of compositions. A composition let's you render a scene in a different way, with a different in and out point, different camera moves, and different rendering styles.

A few tips

  • To create more contrast between positive and negative space, change the 'Simplify' level.
  • To clean the background away from clips, change the Z-threshold
  • The rendering styles are screen-blended onto one another. So you can get a lot of mileage out of combining points,wires, and meshes together and varying opacities.
  • If your interface is getting crowded you can nudge the timeline widths by pulling on the green stripes at the bottom. Shrink them up and the interface will snap back to the bottom giving your render window some more room.

To create a camera track, move the camera to where you'd like to look at the scene and scrub to the point in time you'd like to view. Press "Set camera point" on the GUI or press SHIFT+T. You'll notice a blue marker appear on the Camera timeline. You can then move the timeline and the camera to a new location and press SHIFT+T again. By selecting "Lock to Track" or pressing SHIFT+L the camera will move along this path and is no longer free floating.

You can click and drag the camera points to change their location in time. You can delete them as well with the Delete or backspace key. By clicking on the beginning or end triangle and hitting the up and down arrows you can have the cameras ease in or out (represented by the curved arc), or cut (represented by the small)

The Depth of Field effect is most convincing for creating a virtual camera language. When Draw DOF is selected in the GUI. By setting the Depth Range you can select which distance is in focus. Tweak the range and blur to achieve the desired effect. Depth of field keyframes matched to camera moves can create really cool convincing rack focus effects. The white rectangle on the top left assists you in focusing. The geometry in focus shows up in black, while everything out of focus is white.

Save the composition when you are ready to export or want to experiment with something else. To switch compositions, click the name of the composition at the top and select a new scene.

EXPORTING

Exporting works by rendering the a given composition's camera track into a series of PNG frames that can be compiled into a movie using quicktime or After Effects.

To export, click on the composition name to view all the scenes. Selecting a scene will show all the compositions in that scene, each of which has a small 'R' button next to it. This stands for Render, and by toggling it on you will add this composition to the render queue.

Once you've selected all the compositions you wish to render, click the "Start Render Batch >>" button and sit back and relax as all the compositions you've selected render.

Render frames are 1920x1080 and saved into _RenderBin/ folder underneath the main MediaBin/ (right next to _calibration). Each folder is stamped with the scene, composition, and date so nothing will ever be overwritten.

NOTE: each rendered comp needs a camera track. Weird stuff may happen if you render without one.