
Investigations of Image Fusion
Electrical Engineering and Computer Science Department 
1. Introduction 

1.1 Multisensor Data Fusion 

1.2 Image Fusion 

Improve reliability (by redundant information)
Improve capability


Sensor1  Sensor2  Effect 
TV  IR  Penetration, day/night 
MMW  IR  Penetration, discrimination 
TV  Laser(highpower)  Induced vibration signatures 
IR  UV  Background discrimination 
Multispectral  Panchro  Discrimination features and context 
ALL  Digital terrain map  Discrimination and location 
Laser(pulsed)  TV/IR  Thermal signature 
TV  SAR  Mapping 
2. Review of image fusion research 

2.1 The evolution of image fusion research 

Several types of pyramid decomposition are used or developed for image fusion, such as:
Since then, image fusion began to receive increasing attention. Click to see the number of papers found in recent years using the keyword "image fusion" from the database INSPEC.
2.2 Relevant wavelet theory 

The original space V0 can be decomposed into a lower resolution subspace V1, the difference between V0 and V1 can be represented by the complementary subspace W1. Similarly, we can continue to decompose V1 into V2 and W2. The above graph shows a 3level decomposition. For an Nlevel decomposition, we will obtain N+1 subspaces with one coarsest resolution subspace Vn and N difference subspace Wi, i is from 1 to N. Each digital signal in the space V0 can be decomposed into some components in each subspaces. In many cases, it's much easier to analyze these components rather than analyze the original signal itself.
We can apply a pair of filters to divide the whole frequency band into two subbands, then apply the same procedure recursively to the lowfrequency band on the current stage. Thus, it is possible to use a set of FIR filters to achieve the above multiresolution decomposition. Here is one way to decompose a signal using filter banks:
The filters in different level can be generated iteratively by the following relations:
The only thing we need here is h1 and g1 which correspond to a certain wavelet.
Thus, we obtain a wavelet orthonormal basis:
A discrete signal x can be described by these scaling function and wavelet function:
where s and d are wavelet coefficients.

Pyramid hierarchy of 2D DWT 
After one level of decomposition, there will be four frequency bands, namely LowLow (LL), LowHigh (LH), HighLow (HL) and HighHigh (HH). The next level decomposition is just apply to the LL band of the current decomposition stage, which forms a recursive decomposition procedure. Thus, an Nlevel decomposition will finally have 3N+1 different frequency bands, which include 3N high frequency bands and just one LL frequency band. The 2D DWT will have a pyramid structure shown in the above figure. The frequency bands in higher decomposition levels will have smaller size.
3. Image fusion research in Lehigh 

In our research, we are mainly focusing on the following two topics:
3.1 Image fusion schemes 

Wavelet transform is first performed on each source images, then a fusion decision map is generated based on a set of fusion rules. The fused wavelet coefficient map can be constructed from the wavelet coefficients of the source images according to the fusion decision map. Finally the fused image is obtained by performing the inverse wavelet transform.
From the above diagram, we can see that the fusion rules are playing a very important role during the fusion process. Here are some frequently used fusion rules in the previous work:
When constructing each wavelet coefficient for the fused image. We will have to determine which source image describes this coefficient better. This information will be kept in the fusion decision map. The fusion decision map has the same size as the original image. Each value is the index of the source image which may be more informative on the corresponding wavelet coefficient. Thus, we will actually make decision on each coefficient. There are two frequently used methods in the previous research. In order to make the decision on one of the coefficients of the fused image, one way is to consider the corresponding coefficients in the source images as illustrated by the red pixels. This is called pixelbased fusion rule. The other way is to consider not only the corresponding coefficients, but also their close neighbors, say a 3x3 or 5x5 windows, as illustrated by the blue and shadowing pixels. This is called windowbased fusion rules. This method considered the fact that there usually has high correlation among neighboring pixels.
In our research, we think objects carry the information of interest, each pixel or a small neighboring pixels are just one part of an object. Thus, we proposed a regionbased fusion scheme. When make the decision on each coefficient, we consider not only the corresponding coefficients and their closing neighborhood, but also the regions the coefficients are in. We think the regions represent the objects of interest. We will provide more details of the scheme in the following.
We first apply Canny edge detection on the LL band of the wavelet coefficient maps of the source images. The results are the edge images which provide the location and intensity of edges in the source images. Next, we perform region segmentation using the edge information. The output are region images, in which different values represent different regions. Then, the activity levels of each region is obtained by averaging the highfrequency wavelet coefficients, which may be more informative. Thus, we generate the region activity tables. The larger activity value mean more informative of the region. Base on the edge and region image and the region activity table, we apply the following fusion rules to compute the fusion decision map.
3.2 Image quality and fusion evaluation 

Assessing image fusion performance in a real application is a complicated issue. In particular, objective quality measures of image fusion have not received much attention. Some techniques to blindly estimate image quality are proposed in our research. Such quality measures can be used to guide the fusion and improve the fusion performance.
The observed image equal the signal plus the noise. Here, we are not going to study the image itself, but its edge intensity distributions. Since the edge intensity is come from the gradient information of the image, we consider using a mixture of Gaussian density to approximate the histogram of the gradient values:
Using the relationship between gradient and edge intensity, it's easy to see that we can use a mixture of Rayleigh density to approximate the histogram of edge intensity:
Here, the parameters and weights of mixture terms can be obtained using the EM algorithm. M is the number of terms used in the mixture density. The EM algorithm is a recursive maximumlikelihood estimation of mixture density. Click to see more details.
Q will decrease when i.i.d. Gaussian noise is added and will have the minimum value when the signal is also Gaussian. To compare with the normal SNR measurement, we may define a QR measurement as:
Other types of image degradation such as blurring are also considered in our work. We have pointed out earlier that the the term with largest variance parameter in the mixture model corresponds to the strong edges in the image. Since blurring will influence these strong edges significantly, we consider using the largest term to monitor the blurring. Thus, we may define IQ to estimate the overall image quality:
Here, noise will mainly influence Q and blurring will mainly influence .
The following Gaussian smoothing is used to simulate the blurring effect:
A generalized IQ measurement can be defined as:
The function g1 and g2 will determine the relative importance of blurring and noise. We can adjust g1 and g2 to meet the different requirements in specific applications.
4. Potential applications of image fusion 
