2. FUNDAMENTACIÓN
3.12. Protocolos de Comunicación
In this chapter we have seen that the initial assumption of a fixed focal length during rotation of the laparoscope does not hold. As a consequence, the radial distortion coefficients are also not fixed. However, by fixing the focal length and radial distortion parameters to a single value, and by approximating the principal point and decentering distortion parameters by an ellipse, the translation between the camera coordinate system and coordinate system of the reference sensor is forced to behave as a circular motion in a plane during rotation of the laparoscope. Despite its inaccuracies, the camera model produces a reprojection error in the order of only 1 pixel RMS. The low error and circular motion of the hand-eye translation make the proposed model a good approximation of the camera. Rotation of the hand-eye translation component along the circle is not linear to the rotation between the two parts of the laparoscope. Hand-eye translation rotation angle can be modeled as a function of the scope’s rotation angle using a Fourier sum.
Changing the direction of the scope rotation results in a substantial change of the hand-eye transformations. This is caused by a displacement of the camera coordinate system. The only explanation we see for a displacement of the camera coordinate system is a displacement of the image sensor within the laparoscope. In the model, it was assumed that only motion possible between the two camera components is a rotation around a fixed axis through the two components. This freedom of motion would require a significantly more complex model for the laparoscope. As the pose of the inner compound lens-system cannot be measured in relation to anything accessible on the outside of the laparoscope, it does not seem feasible to create a simple model for the laparoscope using external tracking.
- 63 -
CH6: General discussion and conclusions
In the few literature references available on camera calibration of oblique viewing laparoscopes, several assumptions are made on the behavior of the intrinsic and extrinsic camera parameters. All laparoscope configurations used in these references consist of a conventional laparoscope setup with a camera head and separable optics system. The Olympus EndoEye HD 10 mm laparoscope used during this study is an all-in-one system with the image sensor located in the tip of the laparoscope. To our knowledge, there are no literature references available of a (working) calibration method for oblique viewing laparoscopes with this chip-on-the-tip technology. The behavior of intrinsic and extrinsic camera parameters during rotation of the laparoscope were evaluated to verify if the assumptions made for the conventional laparoscopic setup are still valid.In literature, focal length and radial distortion coefficients are assumed to be fixed, and the principal point is either described as a fixed point, or as a circular motion around a point on the image plane. We have shown that, for our laparoscope, the focal length does change during rotation. Radial distortion therefore also changes as it is a function the focal length. Our observations showed that the principal point behavior can best be modeled as an ellipse instead of a circle.
Decentering distortion is not included in any of the available references. Decentering distortion originates from misalignment of lens elements. As the total lens-system of the laparoscope consists of two compound lens-systems rotating relative to another, decentering distortion also changes during rotation. The estimated decentering parameters describe a magnitude and direction that can also be modeled by an ellipse during rotation. We have shown that the addition of decentering distortion improves the reprojection results of the camera model.
Even though the assumption of a fixed focal length and radial distortion is false, it is used in the designed model as it leads to better extrinsic camera parameters needed for hand-eye calibration. The proposed camera model has a reprojection error on calibration images of 0.5-1 pixel RMS. Applying the camera model to validation images increased the reprojection error to a mean of 1 pixel RMS. This increase can, at least partially, be contributed to errors in estimation of the rotation angle, leading to incorrect intrinsic parameters used to represent the camera during reprojection. An average error in the order of 1 pixel RMS is sufficient for any AR purposes of this laparoscope.
Two optical sensors are used to track the motion of the laparoscope. One is attached to the handle, and the other to the cylinder of the laparoscope. Rotation angle of the laparoscope is determined by solving a non-linear problem to find the center and axis of rotation.
The sensor attached to the cylinder of the laparoscopes is used as reference for the camera coordinate system as it has a fixed relation to the optical axis, and it being closest to the tip of the laparoscope produces smaller tracking errors. If our intrinsic camera model is used, the origin of the camera coordinate system produces a circular motion in the coordinate system of the reference sensor during rotation. The rotation angle of this circular motion was modeled using a Fourier sum to correct for the non-linearity in relation to the laparoscope’s rotation angle. Origin of the camera coordinate system was modeled as a point rotating at a fixed radius around the center point of the circular motion with the axis of rotation normal to the center of the circular motion.
- 64 -
As the direction of the optical axis is fixed in relation to the reference sensor, the orientation of the camera coordinate system could be modeled by a rotation around the optical axis with an angle equal but in opposite direction of the rotation angle of the laparoscope. In the model, the orientation obtained in the default configuration of the laparoscope (oblique view directed downwards) was used as reference for the orientation at other viewing angles of the laparoscope.
The intrinsic camera model and hand-eye translation model have shown to be a good approximation of the truth on the calibration data of multiple laparoscopes. However, the camera orientation model showed hysteresis of rotation, and the overall model could not produce accurate results on validation images. In the validation images, the reprojection error increased to a maximum of 100 pixels RMS. The cause of this error was determined to be motion of the image sensor within the laparoscopes by changing the direction of scope rotation. Movement of the image sensor causes the camera coordinate system to move, making the hand-eye translation model incorrect. This freedom of motion permits changing focal lengths, hysteresis of the rotation, and the observed movements in camera coordinates. Producing an accurate model for the laparoscope requires the pose of both parts of the camera system to be tracked. This was attempted by placing two optical sensors on the laparoscope. The sensor on the handle of the laparoscope was assumed to be fixed in relation to the image sensor, but this assumption is proven to be incorrect. As there is no other place accessible from the outside of the laparoscope to track the image sensor, it does not seem possible to create a model for this specific type of laparoscope based on external tracking. However, none of the laparoscopes was calibrated more than once. Repeating the calibration multiple times, and closer inspection of the behavior of the camera coordinate system, may reveal a pattern in the behavior of the image sensor in relation to rotation of the scope. This pattern can then be used to predict the behavior of the image sensor during rotation. This would require the rotation history of the laparoscope to be included in the model, making it substantially more complex. Inclusion of the rotation history would also require all rotations of the laparoscope to be measured by the optical tracking system, a challenging task that limits the usability in a clinical setting. If we compare our laparoscope to the conventional laparoscopic setups used by other authors, we see that the issues in tracking are specific to our chip-on-the-tip system. In conventional setups, the image sensor is located and fixed within the camera head. This configuration allows accurate pose estimation of the image sensor and lens-systems, and therefore allows the laparoscope to be modeled by external tracking. However, most of the literature references are between 10-15 years old. Since the initial publication, none of the authors have released a follow up publication or shown a clinical application of the system. One of the possible reasons for this is that the issues experienced in our system are also present in the conventional system if used in the clinic. During clinical use of the laparoscope, the scope is introduced into the patient through a trocar that provides the pivot point for changing the laparoscopes view to the general direction of interest. The abdominal wall can be several centimeters thick. Pivoting of the laparoscope in the abdominal wall therefore requires force to be put on the camera head of the conventional laparoscope. This force can cause the camera head to tilt and maybe even move in relation to the optical system, leading to similar issues as experienced in our system. However, as there is no literature available on this topic, this explanation is no more than an assumption based on extrapolation of our results.
All experiments performed on the laparoscope were in a static setting where the acquisition delay between the laparoscope and optical tracking system played no role. Real-time AR application in a dynamic setting requires the delay between both systems to be known. A delay estimation procedure was developed that can accurately estimate the delay. In the procedure, an object is rotated at a
- 65 -
constant angular frequency and tracked by both systems. In the images, the position of the object is estimated with subpixel accuracy using a template and normalized cross-correlation. A sinus was fitted to the constant rotation in both systems and the delay was determined by the phase delay between the two. The system was validated by introducing delays of up to 200 ms to the optical system. The system can estimate the delay accurate up to 5 ms. However, it was not validated if the delay between the two systems could be reduced to zero.