- A rigid motion between the reference mark of the world and the reference mark of the cone,
- The conical reflection,
- A rigid motion between the reference mark of the cone and the reference mark of the camera,
- The perspective projection.
- First of all, on the photo of left the synthetic picture has been calculated, while using data constructors for the different parameters of the model. On this picture no correspondence is noted.
- Then, on the photo of the center the synthetic picture is identical to the first one, but this time, we put in correspondence projections of the apex of the cone. Thus, as you can note it, the radial straight lines correspond. On another way, the projection of the vertical elements corresponds. It is this particularity that is used in the major part of applications using this sensor.
- Finally, on the last photo, the synthetic picture has been calculated, while using values of parameters estimated during the calibration. On this picture the projection of the vertical elements corresponds, but all other projections also.
Abstract
The sensor
The model
The calibration
The localization
Current works
Videos of experimental results
In order to extend the detection to other features that radial straight lines in omnidirectional images, we chose to try to understand the phenomenon of picture formation descended of a sensor made of a CCD camera and a conical mirror.
In this setting I do a thesis that is about the modelling and the calibration of this sensor.
This calibration permits us to know parameters composing the model of picture formation.
With the help of the model and these parameters we programmed a simulator of omnidirectional pictures.
A study on the use of these pictures for localization and possibly the prediction is under realization.
Omnidirectional vision is gotten while placing a black & white CCD camera under a conical mirror, as shows the following image.
The advantage of this sensor type is to get in only one acquirement, a picture that covers a fields of view of 360°. Thus, the gotten pictures present a very interesting characteristic. Indeed, if the optical axis corresponds with the revolution axis of the cone, the parallel striaght lines to these project themselves in a radial way in the picture. In another way, if we place this sensor in an environment, so that the basis of the cone is parallel to soil, the vertical straight lines of the environment will project themselves in radial straight lines in the picture.
Here is an example of what we can get:
 |
The model of camera is well known. On the other hand, the conical reflection is less. Therefore, we used the notion of virtual point. As shows the figure on the right, punctually, we can consider that the conical mirror functions like a plane mirror. Therefore, the projection of the real point P by conical reflection is equivalent to the point V (virtual). So, we are able to modelize the conical reflection.
|
To calibrate we conceived a calibration pattern well adapted to the sensor of which a diagram is visible on the right. This pattern calibration is composed of four plan perpendiculars with a width of 20cm each. As you can note it, every plan is covered of motive in checkerboard in order to facilitate the extraction of points of calibration. Indeed, the vertical straight lines of the pattern are going to project themselves radially and the horizontal straight lines according to portions of ellipsis. So, the extraction of these motives will be easily feasible and we will get a set of calibration point relatively important. Besides, in order to determine the orientation of the pattern calibration, a reference mark has been placed on one of the four plans. The projection of this pattern on the image plan is shown below.
To get the set of points of calibration we must extract coordinates of motive corners in the picture. To do it we apply a Sobel filter on the picture, then a binarisation. Then, with the help of a Hough transform, we do the radial straight lines extraction. Now, the binary picture only includes the horizontal straight lines projection that we are going to extract while interpolating them by portions of ellipses. We only have to calculate intersections between the radial straight lines and portions of ellipses in order to get, with a sub-pixel accuracy, the picture coordinates of calibration points.
The binary image) |
The binary image after radial straight lines extraction) |
Radial straight lines and portions of ellipses) |
The pattern projection and calibration points) |
At this stage we have a set of 3D points and their 2D correspondents on the image plan. We have an over - dimensioned system that we can solve by minimization methods. We chose to use theLevenberg-Marquardt algorithm because it is the method the more frequently used in this kind of resolution.
After the resolution, we have an evaluation of different parameter that composes our mathematical model.
In order to show the interest that present the calibration, here is a small comparative. In this example we calculated the synthetic pictures of the projection of the pattern calibration while using the model presented higher. These pictures have been superimposed to a real picture.
A synthetic picture calculated while using data constructors.) |
The same synthetic picture that on the left, but this time, we made coincide projections of the apex of the cone.) |
A synthetic picture calculated while using parameters estimated during the calibration.) |
As you can see on the three previous photos, the calibration permits to simulate with a high accuracy the process of omnidirectional pictures formation.
Therefore, we can deduct that the model defines correctly represents the phenomenon of acquirement of an omnidirectional picture. Also, we can say that the evaluation of parameters of the system permits to calculate pictures very close to the real pictures (as shows the following picture).
| Real image | Synthetic image | Superimpose |
) |
) |
) |
) |
) |
) |
) |
) |
) |
Considering this precision of projection we tried to do spatial localizations of the sensor while using the model. Until now, localizations made with this sensor used the essential property: the vertical beakons of the environment project themselves in a radial way in the picture. For our part we wanted to use others primitive: points. In fact, we chose to localize while using 12 3D points and their projection in the image plan.
The environment of work (below) has for measurements (2m by 3m5). It is composed of 5 block of 1m25 height and different widths. Every block is covered of a set of motive permitting a strong contour detection. Thus, we think to put in correspondence binary images from panoramic picture.
Once the sensor have been calibrated and an image have been acquire, the user clicks 12 well known points. Thus, we get a set of 12 2D points and their 3D correspondents.
Off line, with the help of the model, we created a basis of projection of these twelve points with a paving all centimeters. While using the Hausdorff distance, we determine what is the position (and orientation) that present best possible correspondence. Thus, we arrange value of initialization for the evaluation of the rigid movement between the reference mark of the world, and the one of the sensor. With this minimization we get a spatial evaluation of the position of the sensor in the environment. The figure below present a planar projection of these results:
)
Results are rather encouraging. The orientation is estimated correctly, with an average mistake lower to 0.3°. In position the average mistake is around one centimeter.
This method of localization seems very interesting, but it presents a defect, it requires the human intervention for the extraction of points in the picture. Therefore, we want to suppress this human " stage " in the localization phaze. To do it, we work on the conception of a method of position evaluation with the help of a basis of synthetic panoramic picture clarified in the following part.
With the mathematical model that we defined, after calibration, we are able to calculate planar projections from omnidirectional pictures. The picture below present some results:
| Omnidirectional image | Planar projection | ||
) |
) |
) |
) |
) |
) |
) |
|
) |
) |
) |
) |
) |
) |
) |
|
) |
) |
) |
) |
) |
) |
) |
|
For the meantime, no work has been made on the use of these planar projections.
In the same way it is possible to calculate a cylindrical projection of omnidirectional picture in order to get some panoramic pictures. Here are some photos:
| An omnidirectional picture The panoramic picture corresponds ) |
An omnidirectional picture) The panoramic picture corresponds ) |
The interest of such picture is that for a given position they are invariant in orientation. Thus, it is possible to determine orientation by doing a simple horizontal scrolling of the picture. That's why the horizontal resolution of the picture is 1440 pixels. Of this way we get a definition in orientation of 0.25°. The vertical resolution is defined in order to preserve a correct ratio (to have a coherent visual aspect).
My present works are about the conception of a method of image matching between a real panoramic picture among a set of synthetic panoramic picture (see below). The goal is to manage to estimate the position of the sensor in its environment with the help of a basis of synthetic panoramic pictures.
An omnidirectional picture) |
A synthetic omnidirectional picture at same position, and orientation ) |
The panoramic pictures correspond) ) | |