Creating Panoramas Using Panorama Tools

H. Dersch

Technical University Furtwangen

der@fh-furtwangen.de

15. July 1998

This document describes the use of the program Panorama_Tools. to create many types of Panoramas. It covers especially the new tool "adjust" (version 1.2 or later) which allows to use any mixture of rectilinear, panoramic, and fisheye images to compose a panorama. These images need not be aligned in any way, and may even be shot handheld (as long as parallax errors are kept small). It includes an automatic positioning tool.

"Panorama_Tools" is a Plug-In for the Macintosh program GraphicConverter, and requires a Power Macintosh. Please see the Readme file in the distribution about how to install it. A Windows version is being worked on.

The panorama being created by "adjust" is always a full spherical view (360° x 180°) in an equirectangular projection even if the input images only cover part of it. You can visualize this as being the equivalent of a world map showing the whole surface of the earth.

This panorama can be used for many types of applications:

LivePicture's spherical RealVR viewer directly digests this format. See my tutorial on Spherical Panoramas.
Apple's QTVR movies require a similar, slightly different projection which can be easily derived from it. See my tutorial on Cylindrical Panoramas.
Any still image (panoramic, rectilinear, or fisheye) can be extracted from the panorama using the "remap" and "perspective" tools. This will be covered in a separate tutorial.

To illustrate the technique we start from the result. The example panorama is a view of the Grand Canyon:

If you have exact control on how your images are oriented within the panorama (eg by using a panorama head and a tripod, and by carefully adjusting the camera to well aligned positions) you may proceed directly to the chapter "Panorama Creation". If some parameters (pitch, yaw, roll, field of view, etc) are unknown, you should start by running the images through the position optimizer.

Position Optimizer

The example panorama of this tutorial was generated using three handheld shots made with a 180° circular fisheye lens:

You are not required to use this type of images: The adjust-tool is capable of combining any mixture of normal, rectilinear images, panoramic images, and fisheye images (full-frame and circular) into a full spherical view.

The optimizer requires the user to specify a set of control points. Prior to that the images should be corrected if necessary: If a barrel-distortion has to be compensated, you should do it before reading the control points etc. If you are worried about image degradation due to multiple sampling steps, you can later combine the found optimized positions and the previous correction steps into one transformation, and apply it to the original scans.

The images of the current example were first cropped to exactly cover the image circle, then remapped to "ideal" fisheye mapping (via the "Sigma" settings file, which works for this lens, a Spiratone adapter, too), and then a center gradient filter of amplitude "20" was applied. This center gradient filter (part of the "correct" tool) turned out to be very important in many panoramas, even if the images showed no visible radial luminance change.

After this preparation you have to identify control points in the images: These are points in the overlap region of neigboring images, which show identical features. This could be a small stone, some edge, a twig etc. For each pair of overlapping images you should identify at least 2-3 points, and notice the x/y - screen-coordinates of the points as displayed by GraphicConverter. The more points you specify, the better will be the fit. Also, fitting many parameters requires more control points than fitting only a few. For the current example I selected three points for each pair of images, resulting in a total of nine points.

The coordinates of the points and some additional information regarding the images has to be written into a small script file to be read by Panorama Tools. This script file is a standard text file which you can generate using any plain text editor (BBEdit, SimpleText, ...). For the current example this file has the following content:

# Script Grand Canyon

p w1800

# Anchor is image 2 
i w900 h900 f2 r0 p0 y60 v180 
i w900 h900 f2 r0 p0 y-60 v=0 
i w900 h900 f2 r0 p0 y-180 v=0 

v y0 y1 

c n0 N1 x142 y528 X611 Y516
c n0 N1 x347 y318 X813 Y294
c n0 N1 x211 y443 X688 Y446

c n0 N2 x875 y348 X99 Y370
c n0 N2 x814 y505 X47 Y538
c n0 N2 x848 y435 X60 Y452

c n1 N2 x104 y552 X696 Y525
c n1 N2 x147 y440 X753 Y431
c n1 N2 x177 y588 X741 Y581

The syntax for this script is straightforward: Only lines beginning with either of p, i, v and c are read, so you might add any amount of comments and other information. All parameters are provided using an identifier and a numerical value, eg. "w300" means width is 300 pixels. The "=" operator links a variable in one image to the corresponding variable in another image. The information is provided in the following format depending on the first letter of a line. Notice that numbering of images starts with 0. All coordinates use the screen system, ie origin (x=0;y=0) is top left.

p - panorama description. Only width (w) required, eg. w1800.
i - input image description. Required is
- w - width in pixels
- h - height in pixels
- f - lens type (0 - rectilinear, 1 - panorama, 2 - fisheye)
- y - yaw, -180...+180, horizontal position of image in panorama (in degrees, may be floating point value).
- p - pitch, -90...+90, vertical position of image in panorama (in degrees, may be floating point value).
- r - roll, rotate image (in degrees, may be floating point value).
- v - horizontal field of view (in degrees, may be floating point value).
v - Variables to be optimized. Specify which variables should be subject to the optimization procedure. Choose any of y (yaw), p (pitch), r (roll), v (hfov). The argument to this letter specifies the number of the image belonging to the variable (starting from 0), eg. "y0" means: "optimize yaw in image number 0". Note that you still have to provide a value for yaw in image 0, which is used as initial guess.
c - Control point description
- n - number of first image containing the control point (starting from 0)
- N - number of second image containing the control point (starting from 0)
- x - x-coordinate of control point in first image
- X - x-coordinate of control point in second image
- y - y-coordinate of control point in first image
- Y - y-coordinate of control point in second image

Walking through the example script:

# Script Grand Canyon

This is a comment line which is ignored by the program.

p w1800

We want to create a panorama 1800 pixels wide. The height is automatically set to 900 pixels.

# Anchor is image 2

Another comment.

i w900 h900 f2 r0 p0 y60 v180 
i w900 h900 f2 r0 p0 y-60 v=0 
i w900 h900 f2 r0 p0 y-180 v=0

Description of input images. All are 900x900 pixels large and are fisheye images (f2). Our best estimates for roll and pitch is 0° (r0, p0), and we attempted to shoot in equal 120° intervals. (y60, y-60, y-180). Horizontal field of view is 180° (v180), and the other two images are identical to image No 0 in this respect (v=0). This latter feature is important if we later optimize the field of view: Even if it turns out to be different than the manufacturer's promise of 180°, it certainly is the same for all three images.

v y0 y1

We want to optimize the yaw angles of images 0 and 1. One of the images has to be fixed: in this case we choose image 2 to become the anchor of the panorama. Even if you have to optimize more than just two parameters, you should first start with fewer ones to improve the first estimates. This makes the optimization more stable.

c n0 N1 x142 y528 X611 Y516

Now follows the list of control points. This one is in the overlap region of image 0 and 1, and has coordinates 142|528 in image 0 vs 611|516 in image 1.

We save this textfile under the name "GC script", and select "Adjust" in the "Effect" menu of GraphicConverter. We push the radio button "Run Optimizer", and find our previously saved script file after pushing "Find Script". Then we push "OK", and GraphicConverter runs the optimizer. The results are written to the file "Optimum" in GraphicConverter's folder. Usually, GraphicConverter should launch SimpleText for you, which displays the results of the calculation. "Optimum" consists of a copy of the input script together with the following information:


====================================================================
Parameters for Each Input Image:
(*) - optimized         (p) - preset 

Image No 0:
Yaw:	27.7187¡ (*)	Pitch:	0¡ (p) 
Roll:	0¡ (p)	HFov:	180¡ (p)
Polynomial Coefficients: a   0.000000 (p); b   0.000000 (p); c   0.000000 (p)
Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)

Image No 1:
Yaw:	-64.1855¡ (*)	Pitch:	0¡ (p) 
Roll:	0¡ (p)	HFov:	180¡ (p)
Polynomial Coefficients: a   0.000000 (p); b   0.000000 (p); c   0.000000 (p)
Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)

Image No 2:
Yaw:	-180¡ (p)	Pitch:	0¡ (p) 
Roll:	0¡ (p)	HFov:	180¡ (p)
Polynomial Coefficients: a   0.000000 (p); b   0.000000 (p); c   0.000000 (p)
Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)

====================================================================
Control Points: Distance between desired and fitted Position (in Pixels)

Control Point No 0:  18.3675
Control Point No 1:  44.1678
Control Point No 2:  17.7343
Control Point No 3:  33.7559
Control Point No 4:  26.4404
Control Point No 5:  29.4265
Control Point No 6:  31.494
Control Point No 7:  28.3659
Control Point No 8:  22.6133

Each image is listed together with the Euler angles (yaw,pitch,roll) and other information describing the position of the image in the panorama. The parameters marked (*) have been optimized. Ignore the "polynomial coefficients" and "horizontal/vertical shift" parameters for the moment. The last list describes the quality of the fit: It is the distance measured in screenpoints between the points of the two images after conversion to a panorama. A perfect fit would yield zero for each point. Obviously, the results are far from being perfect.

The fitted yaw-values differ quite a bit from the initial guesses. To improve the adjustment, we should optimize roll and pitch angles, and also the horizontal field of view. The next attempt uses the same input script with some slight changes:

....
i w900 h900 f2 r0 p0 y27 v180
i w900 h900 f2 r0 p0 y-64 v=0 
i w900 h900 f2 r0 p0 y-180 v=0 

v y0 y1 r0 r1 p0 p1 v0
....

We have entered the previously found yaw values as new starting points, and we entered all additional variables in the "v"-line. We do not optimize image 2, because one image has to stay fixed. This time we can simply push "OK" after selecting "adjust" since "Panorama_Tools" remembers the location of the script file.

Before running the optimizer again, be sure to close the file "Optimum" in SimpleText, because GraphicConverter can not write to an open file. The results of the second optimization are much better now:


====================================================================
Parameters for Each Input Image:
(*) - optimized         (p) - preset 

Image No 0:
Yaw:	29.2614¡ (*)	Pitch:	-7.72616¡ (*) 
Roll:	0.844557¡ (*)	HFov:	172.591¡ (*)
Polynomial Coefficients: a   0.000000 (p); b   0.000000 (p); c   0.000000 (p)
Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)

Image No 1:
Yaw:	-63.5094¡ (*)	Pitch:	-2.4904¡ (*) 
Roll:	3.55895¡ (*)	HFov:	172.591¡ (*)
Polynomial Coefficients: a   0.000000 (p); b   0.000000 (p); c   0.000000 (p)
Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)

Image No 2:
Yaw:	-180¡ (p)	Pitch:	0¡ (p) 
Roll:	0¡ (p)	HFov:	172.591¡ (*)
Polynomial Coefficients: a   0.000000 (p); b   0.000000 (p); c   0.000000 (p)
Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)

====================================================================
Control Points: Distance between desired and fitted Position (in Pixels)

Control Point No 0:  3.12264
Control Point No 1:  5.89959
Control Point No 2:  5.79277
Control Point No 3:  2.01965
Control Point No 4:  3.57084
Control Point No 5:  2.30948
Control Point No 6:  3.88106
Control Point No 7:  2.5718
Control Point No 8:  1.69617

Interestingly, the horizontal field of view is smaller than 180°, a result which I consistently also get for the Olympus 8mm. Differences of 2-5 pixels are usually quite acceptable in a 2000 pixel wide panorama. So we save and use the found data for the creation of the panorama.

If your optimization does not converge, you should check the following possible reasons:

Initial guess too far off; try some other values.
Too few control points.
Too many parameters to optimize
Typing errors for the control point coordinates (most frequent cause in my case).

Panorama Creation

The "Create Panorama" option of the "adjust"-tool creates a full spherical panorama from each input image. To set the options, push the button "Options", and the following dialog appears:

The upper portion specifies the source image characteristics. Most important here is the type of lens you used, in this case a fisheye. The program distinguishes between circular and full frame fisheyes, because in the latter case the whole image content is valid and used for calculation, whereas the former only uses a circle. Next is the horizontal field of view, to be entered in degrees. For the 8mm fisheye we choose 172.6°, the value we found in the optimization. If you don't know the value, you can choose from the following list:

7.5-8mm fisheyes usually have around 180°.
16mm full frame fisheyes have 135-140° (landscape) or 90-95° (portrait)
Rectilinear lenses for 35mm equipment can be calculated using the formula
HFOV = 2 x arctan (18mm / focal length) for landscape mode
HFOV = 2 x arctan (12mm / focal length) for portrait mode

Notice that you have to recalculate HFOV, if you cropped your image!

If you have managed the "correct" tool and found a suitable correction setting for your lens/scanner set up, you can specify that by selecting the correct button. The program will then include the correction into the transformation.

The lower portion specifies the characteristics of the to be created panorama. "Width" is specified in pixels, "Height" is internally set to Width/2. In principle, these parameters can be set completely independant of source image size: all adjustments and scaling is done internally. To preserve image resolution, you should roughly satisfy the equation

Width of Panorama / 360 = Width of source image / HFOV

However, you may enlarge the image to any size: Panorama_Tools employs a high quality interpolator resulting in quite good enlargements. Paradoxically , you might run into troubles, if you create very small panoramas. Reason is the lack of built-in antialiasing, which might result in moiree-effects. If you want to create a small panorama from a large image, I suggest to first decrease image size using GraphicConverter's (or Photoshop's) "scale" tool on the source image.

The fields "roll", "pitch", "yaw" specify how to position the source image in the panorama. Enter -180° < yaw < +180° to position the image horizontally. Enter -90° < pitch < 90° to position the image vertically, and enter any angle for roll to rotate the image around its axis. For the current image we choose the values of our optimizations. After applying this tool to all three input images, we end up with three full sized panoramas:

These panoramas contain an alpha-channel indicating valid (opaque) portions of the image. In case you want to use the alpha channel for further processing (eg as mask in Photoshop) you should save the image in a way to preserve it: In GraphicConverter, choose "Add alpha channel" to make GraphicConverter aware that there are data, then save the image in a format preserving the alpha-channel, eg. Photoshop's file format.

Notice that images with large pitch angles get spread over the entire horizontal axis. This is the reason we created a complete panorama for each input image, since otherwise we would loose some portions of the images.

Final Steps

The final assembly of the panorama has to be done outside Panorama Tools: any graphic program with a feather tool (eg Photoshop) is suitable. Simply cut and paste the pieces together, taking advantage of "natural" seams. A feather of 10 to 20 points usually makes the remaining differences invisible. Usually some retouching is required, eg to fill regions not covered by input images (zenith in current example)

You can view the Grand Canyon image both as spherical RealVR or Cylindrical QTVR Panorama.

For the mathematically interested user: the program employs the Levenberg-Marquardt solver of the MINPACK program. You can find some more information in the file "stdout" which the solver writes into GraphicConverter's directory.

Capturé par MemoWeb à partir de http://www.fh-furtwangen.de/%7Edersch/Adjust/Adjust.html le 30/05/01