1Camera calibration With OpenCV {#tutorial_camera_calibration} 2============================== 3 4Cameras have been around for a long-long time. However, with the introduction of the cheap *pinhole* 5cameras in the late 20th century, they became a common occurrence in our everyday life. 6Unfortunately, this cheapness comes with its price: significant distortion. Luckily, these are 7constants and with a calibration and some remapping we can correct this. Furthermore, with 8calibration you may also determine the relation between the camera's natural units (pixels) and the 9real world units (for example millimeters). 10 11Theory 12------ 13 14For the distortion OpenCV takes into account the radial and tangential factors. For the radial 15factor one uses the following formula: 16 17\f[x_{distorted} = x( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6) \\ 18y_{distorted} = y( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6)\f] 19 20So for an undistorted pixel point at \f$(x,y)\f$ coordinates, its position on the distorted image 21will be \f$(x_{distorted} y_{distorted})\f$. The presence of the radial distortion manifests in form 22of the "barrel" or "fish-eye" effect. 23 24Tangential distortion occurs because the image taking lenses are not perfectly parallel to the 25imaging plane. It can be represented via the formulas: 26 27\f[x_{distorted} = x + [ 2p_1xy + p_2(r^2+2x^2)] \\ 28y_{distorted} = y + [ p_1(r^2+ 2y^2)+ 2p_2xy]\f] 29 30So we have five distortion parameters which in OpenCV are presented as one row matrix with 5 31columns: 32 33\f[distortion\_coefficients=(k_1 \hspace{10pt} k_2 \hspace{10pt} p_1 \hspace{10pt} p_2 \hspace{10pt} k_3)\f] 34 35Now for the unit conversion we use the following formula: 36 37\f[\left [ \begin{matrix} x \\ y \\ w \end{matrix} \right ] = \left [ \begin{matrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{matrix} \right ] \left [ \begin{matrix} X \\ Y \\ Z \end{matrix} \right ]\f] 38 39Here the presence of \f$w\f$ is explained by the use of homography coordinate system (and \f$w=Z\f$). The 40unknown parameters are \f$f_x\f$ and \f$f_y\f$ (camera focal lengths) and \f$(c_x, c_y)\f$ which are the optical 41centers expressed in pixels coordinates. If for both axes a common focal length is used with a given 42\f$a\f$ aspect ratio (usually 1), then \f$f_y=f_x*a\f$ and in the upper formula we will have a single focal 43length \f$f\f$. The matrix containing these four parameters is referred to as the *camera matrix*. While 44the distortion coefficients are the same regardless of the camera resolutions used, these should be 45scaled along with the current resolution from the calibrated resolution. 46 47The process of determining these two matrices is the calibration. Calculation of these parameters is 48done through basic geometrical equations. The equations used depend on the chosen calibrating 49objects. Currently OpenCV supports three types of objects for calibration: 50 51- Classical black-white chessboard 52- Symmetrical circle pattern 53- Asymmetrical circle pattern 54 55Basically, you need to take snapshots of these patterns with your camera and let OpenCV find them. 56Each found pattern results in a new equation. To solve the equation you need at least a 57predetermined number of pattern snapshots to form a well-posed equation system. This number is 58higher for the chessboard pattern and less for the circle ones. For example, in theory the 59chessboard pattern requires at least two snapshots. However, in practice we have a good amount of 60noise present in our input images, so for good results you will probably need at least 10 good 61snapshots of the input pattern in different positions. 62 63Goal 64---- 65 66The sample application will: 67 68- Determine the distortion matrix 69- Determine the camera matrix 70- Take input from Camera, Video and Image file list 71- Read configuration from XML/YAML file 72- Save the results into XML/YAML file 73- Calculate re-projection error 74 75Source code 76----------- 77 78You may also find the source code in the `samples/cpp/tutorial_code/calib3d/camera_calibration/` 79folder of the OpenCV source library or [download it from here 80](https://github.com/Itseez/opencv/tree/master/samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp). The program has a 81single argument: the name of its configuration file. If none is given then it will try to open the 82one named "default.xml". [Here's a sample configuration file 83](https://github.com/Itseez/opencv/tree/master/samples/cpp/tutorial_code/calib3d/camera_calibration/in_VID5.xml) in XML format. In the 84configuration file you may choose to use camera as an input, a video file or an image list. If you 85opt for the last one, you will need to create a configuration file where you enumerate the images to 86use. Here's [an example of this ](https://github.com/Itseez/opencv/tree/master/samples/cpp/tutorial_code/calib3d/camera_calibration/VID5.xml). 87The important part to remember is that the images need to be specified using the absolute path or 88the relative one from your application's working directory. You may find all this in the samples 89directory mentioned above. 90 91The application starts up with reading the settings from the configuration file. Although, this is 92an important part of it, it has nothing to do with the subject of this tutorial: *camera 93calibration*. Therefore, I've chosen not to post the code for that part here. Technical background 94on how to do this you can find in the @ref tutorial_file_input_output_with_xml_yml tutorial. 95 96Explanation 97----------- 98 99-# **Read the settings** 100 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp file_read 101 102 For this I've used simple OpenCV class input operation. After reading the file I've an 103 additional post-processing function that checks validity of the input. Only if all inputs are 104 good then *goodInput* variable will be true. 105 106-# **Get next input, if it fails or we have enough of them - calibrate** 107 108 After this we have a big 109 loop where we do the following operations: get the next image from the image list, camera or 110 video file. If this fails or we have enough images then we run the calibration process. In case 111 of image we step out of the loop and otherwise the remaining frames will be undistorted (if the 112 option is set) via changing from *DETECTION* mode to the *CALIBRATED* one. 113 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp get_input 114 For some cameras we may need to flip the input image. Here we do this too. 115 116-# **Find the pattern in the current input** 117 118 The formation of the equations I mentioned above aims 119 to finding major patterns in the input: in case of the chessboard this are corners of the 120 squares and for the circles, well, the circles themselves. The position of these will form the 121 result which will be written into the *pointBuf* vector. 122 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp find_pattern 123 Depending on the type of the input pattern you use either the @ref cv::findChessboardCorners or 124 the @ref cv::findCirclesGrid function. For both of them you pass the current image and the size 125 of the board and you'll get the positions of the patterns. Furthermore, they return a boolean 126 variable which states if the pattern was found in the input (we only need to take into account 127 those images where this is true!). 128 129 Then again in case of cameras we only take camera images when an input delay time is passed. 130 This is done in order to allow user moving the chessboard around and getting different images. 131 Similar images result in similar equations, and similar equations at the calibration step will 132 form an ill-posed problem, so the calibration will fail. For square images the positions of the 133 corners are only approximate. We may improve this by calling the @ref cv::cornerSubPix function. 134 It will produce better calibration result. After this we add a valid inputs result to the 135 *imagePoints* vector to collect all of the equations into a single container. Finally, for 136 visualization feedback purposes we will draw the found points on the input image using @ref 137 cv::findChessboardCorners function. 138 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp pattern_found 139-# **Show state and result to the user, plus command line control of the application** 140 141 This part shows text output on the image. 142 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp output_text 143 If we ran calibration and got camera's matrix with the distortion coefficients we may want to 144 correct the image using @ref cv::undistort function: 145 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp output_undistorted 146 Then we show the image and wait for an input key and if this is *u* we toggle the distortion removal, 147 if it is *g* we start again the detection process, and finally for the *ESC* key we quit the application: 148 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp await_input 149-# **Show the distortion removal for the images too** 150 151 When you work with an image list it is not 152 possible to remove the distortion inside the loop. Therefore, you must do this after the loop. 153 Taking advantage of this now I'll expand the @ref cv::undistort function, which is in fact first 154 calls @ref cv::initUndistortRectifyMap to find transformation matrices and then performs 155 transformation using @ref cv::remap function. Because, after successful calibration map 156 calculation needs to be done only once, by using this expanded form you may speed up your 157 application: 158 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp show_results 159 160The calibration and save 161------------------------ 162 163Because the calibration needs to be done only once per camera, it makes sense to save it after a 164successful calibration. This way later on you can just load these values into your program. Due to 165this we first make the calibration, and if it succeeds we save the result into an OpenCV style XML 166or YAML file, depending on the extension you give in the configuration file. 167 168Therefore in the first function we just split up these two processes. Because we want to save many 169of the calibration variables we'll create these variables here and pass on both of them to the 170calibration and saving function. Again, I'll not show the saving part as that has little in common 171with the calibration. Explore the source file in order to find out how and what: 172@snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp run_and_save 173We do the calibration with the help of the @ref cv::calibrateCamera function. It has the following 174parameters: 175 176- The object points. This is a vector of *Point3f* vector that for each input image describes how 177 should the pattern look. If we have a planar pattern (like a chessboard) then we can simply set 178 all Z coordinates to zero. This is a collection of the points where these important points are 179 present. Because, we use a single pattern for all the input images we can calculate this just 180 once and multiply it for all the other input views. We calculate the corner points with the 181 *calcBoardCornerPositions* function as: 182 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp board_corners 183 And then multiply it as: 184 @code{.cpp} 185 vector<vector<Point3f> > objectPoints(1); 186 calcBoardCornerPositions(s.boardSize, s.squareSize, objectPoints[0], s.calibrationPattern); 187 objectPoints.resize(imagePoints.size(),objectPoints[0]); 188 @endcode 189- The image points. This is a vector of *Point2f* vector which for each input image contains 190 coordinates of the important points (corners for chessboard and centers of the circles for the 191 circle pattern). We have already collected this from @ref cv::findChessboardCorners or @ref 192 cv::findCirclesGrid function. We just need to pass it on. 193- The size of the image acquired from the camera, video file or the images. 194- The camera matrix. If we used the fixed aspect ratio option we need to set \f$f_x\f$: 195 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp fixed_aspect 196- The distortion coefficient matrix. Initialize with zero. 197 @code{.cpp} 198 distCoeffs = Mat::zeros(8, 1, CV_64F); 199 @endcode 200- For all the views the function will calculate rotation and translation vectors which transform 201 the object points (given in the model coordinate space) to the image points (given in the world 202 coordinate space). The 7-th and 8-th parameters are the output vector of matrices containing in 203 the i-th position the rotation and translation vector for the i-th object point to the i-th 204 image point. 205- The final argument is the flag. You need to specify here options like fix the aspect ratio for 206 the focal length, assume zero tangential distortion or to fix the principal point. 207@code{.cpp} 208double rms = calibrateCamera(objectPoints, imagePoints, imageSize, cameraMatrix, 209 distCoeffs, rvecs, tvecs, s.flag|CV_CALIB_FIX_K4|CV_CALIB_FIX_K5); 210@endcode 211- The function returns the average re-projection error. This number gives a good estimation of 212 precision of the found parameters. This should be as close to zero as possible. Given the 213 intrinsic, distortion, rotation and translation matrices we may calculate the error for one view 214 by using the @ref cv::projectPoints to first transform the object point to image point. Then we 215 calculate the absolute norm between what we got with our transformation and the corner/circle 216 finding algorithm. To find the average error we calculate the arithmetical mean of the errors 217 calculated for all the calibration images. 218 @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp compute_errors 219 220Results 221------- 222 223Let there be [this input chessboard pattern ](pattern.png) which has a size of 9 X 6. I've used an 224AXIS IP camera to create a couple of snapshots of the board and saved it into VID5 directory. I've 225put this inside the `images/CameraCalibration` folder of my working directory and created the 226following `VID5.XML` file that describes which images to use: 227@code{.xml} 228<?xml version="1.0"?> 229<opencv_storage> 230<images> 231images/CameraCalibration/VID5/xx1.jpg 232images/CameraCalibration/VID5/xx2.jpg 233images/CameraCalibration/VID5/xx3.jpg 234images/CameraCalibration/VID5/xx4.jpg 235images/CameraCalibration/VID5/xx5.jpg 236images/CameraCalibration/VID5/xx6.jpg 237images/CameraCalibration/VID5/xx7.jpg 238images/CameraCalibration/VID5/xx8.jpg 239</images> 240</opencv_storage> 241@endcode 242Then passed `images/CameraCalibration/VID5/VID5.XML` as an input in the configuration file. Here's a 243chessboard pattern found during the runtime of the application: 244 245 246 247After applying the distortion removal we get: 248 249 250 251The same works for [this asymmetrical circle pattern ](acircles_pattern.png) by setting the input 252width to 4 and height to 11. This time I've used a live camera feed by specifying its ID ("1") for 253the input. Here's, how a detected pattern should look: 254 255 256 257In both cases in the specified output XML/YAML file you'll find the camera and distortion 258coefficients matrices: 259@code{.xml} 260<camera_matrix type_id="opencv-matrix"> 261<rows>3</rows> 262<cols>3</cols> 263<dt>d</dt> 264<data> 265 6.5746697944293521e+002 0. 3.1950000000000000e+002 0. 266 6.5746697944293521e+002 2.3950000000000000e+002 0. 0. 1.</data></camera_matrix> 267<distortion_coefficients type_id="opencv-matrix"> 268<rows>5</rows> 269<cols>1</cols> 270<dt>d</dt> 271<data> 272 -4.1802327176423804e-001 5.0715244063187526e-001 0. 0. 273 -5.7843597214487474e-001</data></distortion_coefficients> 274@endcode 275Add these values as constants to your program, call the @ref cv::initUndistortRectifyMap and the 276@ref cv::remap function to remove distortion and enjoy distortion free inputs for cheap and low 277quality cameras. 278 279You may observe a runtime instance of this on the [YouTube 280here](https://www.youtube.com/watch?v=ViPN810E0SU). 281 282\htmlonly 283<div align="center"> 284<iframe title=" Camera calibration With OpenCV - Chessboard or asymmetrical circle pattern." width="560" height="349" src="http://www.youtube.com/embed/ViPN810E0SU?rel=0&loop=1" frameborder="0" allowfullscreen align="middle"></iframe> 285</div> 286\endhtmlonly 287
