Investigations of Image Fusion

Electrical Engineering and Computer Science Department
Lehigh University, Bethlehem, PA 18015

1. Introduction
1.1 Multisensor Data Fusion

1.2 Image Fusion

Examples of image fusion combination




TVIRPenetration, day/night
MMWIRPenetration, discrimination
TVLaser(high-power)Induced vibration signatures
IRUVBackground discrimination
Multi-spectralPanchroDiscrimination features and context
ALLDigital terrain mapDiscrimination and location
Laser(pulsed)TV/IRThermal signature

2. Review of image fusion research

2.1 The evolution of image fusion research

Simple image fusion attempts
Pyramid-decomposition-based image fusion
Wavelet-transform-based image fusion

In our research, we are mainly focusing on wavelet based image fusion approaches. An brief review of relevant wavelet theory will be useful for a better understanding of our schemes.

2.2 Relevant wavelet theory

Multiresolution analysis

The wavelet transform is a powerful tool for multiresolution analysis. The multiresolution analysis requires a set of nested multiresolotion sub-spaces as illustrated in the following figure:

Nested multiresolution spaces

The original space V0 can be decomposed into a lower resolution sub-space V1, the difference between V0 and V1 can be represented by the complementary sub-space W1. Similarly, we can continue to decompose V1 into V2 and W2. The above graph shows a 3-level decomposition. For an N-level decomposition, we will obtain N+1 sub-spaces with one coarsest resolution sub-space Vn and N difference sub-space Wi, i is from 1 to N. Each digital signal in the space V0 can be decomposed into some components in each sub-spaces. In many cases, it's much easier to analyze these components rather than analyze the original signal itself.

Filter bank analysis

The corresponding representation in frequency space is intuitively shown in the following graph:

Multiresolution frequency bands

We can apply a pair of filters to divide the whole frequency band into two subbands, then apply the same procedure recursively to the low-frequency band on the current stage. Thus, it is possible to use a set of FIR filters to achieve the above multiresolution decomposition. Here is one way to decompose a signal using filter banks:

Filter bank analysis tree

The filters in different level can be generated iteratively by the following relations:

Iterative generation of filters

The only thing we need here is h1 and g1 which correspond to a certain wavelet.

Discrete wavelet transform (DWT)

The sequence of FIR filters hi and gi cab be used to construct a wavelet decomposition of signal. For this purpose, we can define the discrete scaling function and the discrete wavelet function as:

Discrete scaling function and wavelet function

Thus, we obtain a wavelet orthonormal basis:

wavelet orthonormal basis

A discrete signal x can be described by these scaling function and wavelet function:

1-D discrete wavelet decomposition

where s and d are wavelet coefficients.


Since image is 2-D signal, we will mainly focus on the 2-D wavelet transforms. The following figures show the structures of 2-D DWT with 3 decomposition levels:

Pyramid hierarchy of 2-D DWT

After one level of decomposition, there will be four frequency bands, namely Low-Low (LL), Low-High (LH), High-Low (HL) and High-High (HH). The next level decomposition is just apply to the LL band of the current decomposition stage, which forms a recursive decomposition procedure. Thus, an N-level decomposition will finally have 3N+1 different frequency bands, which include 3N high frequency bands and just one LL frequency band. The 2-D DWT will have a pyramid structure shown in the above figure. The frequency bands in higher decomposition levels will have smaller size.

3. Image fusion research in Lehigh

In our research, we are mainly focusing on the following two topics:

3.1 Image fusion schemes

3.1.1 Image fusion using wavelet transform

The block diagram of a generic wavelet-based image fusion scheme is shown in the following figure:

Block diagram of a generic wavelet-based image fusion approach

Wavelet transform is first performed on each source images, then a fusion decision map is generated based on a set of fusion rules. The fused wavelet coefficient map can be constructed from the wavelet coefficients of the source images according to the fusion decision map. Finally the fused image is obtained by performing the inverse wavelet transform.

From the above diagram, we can see that the fusion rules are playing a very important role during the fusion process. Here are some frequently used fusion rules in the previous work:

Frequently used fusion rules

When constructing each wavelet coefficient for the fused image. We will have to determine which source image describes this coefficient better. This information will be kept in the fusion decision map. The fusion decision map has the same size as the original image. Each value is the index of the source image which may be more informative on the corresponding wavelet coefficient. Thus, we will actually make decision on each coefficient. There are two frequently used methods in the previous research. In order to make the decision on one of the coefficients of the fused image, one way is to consider the corresponding coefficients in the source images as illustrated by the red pixels. This is called pixel-based fusion rule. The other way is to consider not only the corresponding coefficients, but also their close neighbors, say a 3x3 or 5x5 windows, as illustrated by the blue and shadowing pixels. This is called window-based fusion rules. This method considered the fact that there usually has high correlation among neighboring pixels.

In our research, we think objects carry the information of interest, each pixel or a small neighboring pixels are just one part of an object. Thus, we proposed a region-based fusion scheme. When make the decision on each coefficient, we consider not only the corresponding coefficients and their closing neighborhood, but also the regions the coefficients are in. We think the regions represent the objects of interest. We will provide more details of the scheme in the following.

3.1.2 Region-based image fusion

Here are some key points of region-based image fusion approach:

Here is the way to create the fusion decision map:

Data flow for creating the decision map

We first apply Canny edge detection on the LL band of the wavelet coefficient maps of the source images. The results are the edge images which provide the location and intensity of edges in the source images. Next, we perform region segmentation using the edge information. The output are region images, in which different values represent different regions. Then, the activity levels of each region is obtained by averaging the high-frequency wavelet coefficients, which may be more informative. Thus, we generate the region activity tables. The larger activity value mean more informative of the region. Base on the edge and region image and the region activity table, we apply the following fusion rules to compute the fusion decision map.

Base on this fusion decision map, we can construct the fused wavelet coefficient map, then obtain the fusion image by inverse wavelet transform. Here are some image fusion examples:

3.1.3 Image fusion examples

3.2 Image quality and fusion evaluation

Assessing image fusion performance in a real application is a complicated issue. In particular, objective quality measures of image fusion have not received much attention. Some techniques to blindly estimate image quality are proposed in our research. Such quality measures can be used to guide the fusion and improve the fusion performance.

3.2.1 Histogram modeling

Here is the image model we used:

Noisy image model

The observed image equal the signal plus the noise. Here, we are not going to study the image itself, but its edge intensity distributions. Since the edge intensity is come from the gradient information of the image, we consider using a mixture of Gaussian density to approximate the histogram of the gradient values:

Mixture of Gaussian pdf for gradient histogram

Using the relationship between gradient and edge intensity, it's easy to see that we can use a mixture of Rayleigh density to approximate the histogram of edge intensity:

Mixture of Rayleigh pdf for edge intensity histogram

Here, the parameters and weights of mixture terms can be obtained using the EM algorithm. M is the number of terms used in the mixture density. The EM algorithm is a recursive maximum-likelihood estimation of mixture density. Click to see more details.

Click to see some histogram modeling examples.

3.2.2 Noise estimation

Click to see the experimental results

3.2.3 Quality estimation

A more general noise measurement Q can be defined as:

Q will decrease when i.i.d. Gaussian noise is added and will have the minimum value when the signal is also Gaussian. To compare with the normal SNR measurement, we may define a QR measurement as:

Click to see the experimental comparison between QR and SNR measurements

Other types of image degradation such as blurring are also considered in our work. We have pointed out earlier that the the term with largest variance parameter in the mixture model corresponds to the strong edges in the image. Since blurring will influence these strong edges significantly, we consider using the largest term to monitor the blurring. Thus, we may define IQ to estimate the overall image quality:

Here, noise will mainly influence Q and blurring will mainly influence .

The following Gaussian smoothing is used to simulate the blurring effect:

Click to see the experimental comparison between IQ and SNR measurements

A generalized IQ measurement can be defined as:

The function g1 and g2 will determine the relative importance of blurring and noise. We can adjust g1 and g2 to meet the different requirements in specific applications.

4. Potential applications of image fusion