Kinect Registration    

Calibration Target

Though the initial code base could handle more than 1 kinect, we tested with only two at a time. In the calibration target based approach, an asymmetric chessboard was placed such that it was visible to both the kinects. Opencv's chessboard detection code was used to detect it in the image data available from the kinect. Since, the chessboard was asymmetric the order in which the chessboard points were detected was always the same. This results in us obtaining immediate 1-1 correspondence between image pixels. The point cloud streamed by the OpenNI kinect driver has a one to one correspondence with pixel points, with nan values filling those points where the kinect failed to estimate the depth. The image pixel correspondences across the two kinects, can now be easily propograted to correspondences between 3D points. With a sufficient number of correspondences we can compute the rigid transformation required.

Following the same approach as used by Dieter Fox et. al., we found SIFT features for the image points obtained. Using euclidean distance as the distance metric and RANSAC as the sample consensus approach, we choose corresponding image points that are spatially aligned. Here the spatial alignment was with respect to alighning the corresponding 3D points. The least error transformation was chosen as the desired rigid transform. We also experimented with 3D features namely, FHFP and PFH.

In an attempt to improve the results, we wanted to use both shape and color information. FPFH and PFH features use only shape information, and SIFT based feature matching uses only color information. The overall aim was to match pathes of point cloud rather than mere points. For this matching both color (eg. color histogram) and shape information could be used. Since, working with general patches is very challenging, we started with planes and hence I have titled this approach as plane matching. This approach is novel and involves three steps:-

 Plane extraction

Plane extraction

Point cloud library implements a ransac based plane extraction algorithm. This returns the plane that fits the largest number of points in the supplied point cloud. However, this does not discriminate based on the distance between the points. In other words, we get multiple patches of points lying on a plane rather than one patch alone. Using muliple patches as one plane at a time makes is difficult to reply on color matching. Also, not all of these patches may exist in the other kinect's point cloud. Hence, we tried to obtain only one patch per plane. This was achieved by a recursive implemtation of spatial clustering of points and plane fitting within clusters. The recusion was required because a plane fit and plane extract (pulling out of points that lie on that plane) operation, in general breaks the cluster into multiple clusters some of which could be planar.

 Plane description

Each plane is described by its color histogram, which is obained from the RGB values corresponding to the points that lie on the plane. The plane is also described by the coefficients that form its equation. This constitutes the shape part of the description of a plane.

 Plane matching and transform calculation

Plane matching - an illustration

The rotation came right

Rotation estimation
Rotation is obtained by aligning the normals. To make the plane detection robust we restricted ourselves to work with planes having atleast 1000 points. In a general scene we get ~10 good planes and these can be used for plane matching and transform calculation. Since, the number of planes is very small a brute force approach will also work. We search for the least error set of 4 corresponding planes and use these to calculate the transformation. The error metric is defined based on the cosine distance between the color histogram and the angle between the normals after rotation. Ideally the normals should exactly align but error in Kinect data and normal estimation leads to an error of ≤ 6 degress. We thresholded for these errors individually and choose the minimum of those which less than the threshold.
Translation estimation
Translation required an additional distance metric. To estimate the translation the candidate plane correspondences were used in the following manner:-

1. We traversed the set of correspondences and for each correspondence we computed the displacement along the aligned normal.
2. This displacement was added to the translation vector and subtracted (vector subtraction) from the other similar displacements between the remaining corresponding planes.
3. At this end of this process, the net translation vector was used to translate the already rotated planes.

The distance based error metric was further defined as the sum of any residual displacements, along their aligned normals, between corresponding planes. This was weighed and added to the original error metric. This process, however, does not ensure that translations for all three degrees of freedom are known. The correspondences may all end up being between parallel planes and the translation will end up being partially determined. To solve this problem, it was ensured that the normals of the planes vary substantially in their direction. The normals were treated as points on a unit sphere and their covariance matrix was calculated. The eigen values of this matrix determine the spread of these normals along three distinct directions (eigen vectors). The entire process was carried out iff the three eigen values were all greater than a threshold (in our case 0.1).