|
|
Investigations of Image Fusion
Electrical Engineering and Computer Science Department |
| 1. Introduction |
|---|
| 1.1 Multisensor Data Fusion |
|---|
to see a system involving different types of multisensor fusion. | 1.2 Image Fusion |
|---|
|
Improve reliability (by redundant information)
Improve capability
|
 |
Sensor1 | Sensor2 | Effect |
| TV | IR | Penetration, day/night |
| MMW | IR | Penetration, discrimination |
| TV | Laser(high-power) | Induced vibration signatures |
| IR | UV | Background discrimination |
| Multi-spectral | Panchro | Discrimination features and context |
| ALL | Digital terrain map | Discrimination and location |
| Laser(pulsed) | TV/IR | Thermal signature |
| TV | SAR | Mapping |
| 2. Review of image fusion research |
|---|
| 2.1 The evolution of image fusion research |
|---|
Several types of pyramid decomposition are used or developed for image fusion, such as:
Since then, image fusion began to receive increasing attention. Click 
to see the number of papers found in recent years using the keyword "image fusion" from the database INSPEC.
| 2.2 Relevant wavelet theory |
|---|
The original space V0 can be decomposed into a lower resolution sub-space V1, the difference between V0 and V1 can be represented by the complementary sub-space W1. Similarly, we can continue to decompose V1 into V2 and W2. The above graph shows a 3-level decomposition. For an N-level decomposition, we will obtain N+1 sub-spaces with one coarsest resolution sub-space Vn and N difference sub-space Wi, i is from 1 to N. Each digital signal in the space V0 can be decomposed into some components in each sub-spaces. In many cases, it's much easier to analyze these components rather than analyze the original signal itself.
We can apply a pair of filters to divide the whole frequency band into two subbands, then apply the same procedure recursively to the low-frequency band on the current stage. Thus, it is possible to use a set of FIR filters to achieve the above multiresolution decomposition. Here is one way to decompose a signal using filter banks:
The filters in different level can be generated iteratively by the following relations:
The only thing we need here is h1 and g1 which correspond to a certain wavelet.
Thus, we obtain a wavelet orthonormal basis:
A discrete signal x can be described by these scaling function and wavelet function:
where s and d are wavelet coefficients.
 |
Pyramid hierarchy of 2-D DWT |
After one level of decomposition, there will be four frequency bands, namely Low-Low (LL), Low-High (LH), High-Low (HL) and High-High (HH). The next level decomposition is just apply to the LL band of the current decomposition stage, which forms a recursive decomposition procedure. Thus, an N-level decomposition will finally have 3N+1 different frequency bands, which include 3N high frequency bands and just one LL frequency band. The 2-D DWT will have a pyramid structure shown in the above figure. The frequency bands in higher decomposition levels will have smaller size.
| 3. Image fusion research in Lehigh |
|---|
In our research, we are mainly focusing on the following two topics:
| 3.1 Image fusion schemes |
|---|
Wavelet transform is first performed on each source images, then a fusion decision map is generated based on a set of fusion rules. The fused wavelet coefficient map can be constructed from the wavelet coefficients of the source images according to the fusion decision map. Finally the fused image is obtained by performing the inverse wavelet transform.
From the above diagram, we can see that the fusion rules are playing a very important role during the fusion process. Here are some frequently used fusion rules in the previous work:
When constructing each wavelet coefficient for the fused image. We will have to determine which source image describes this coefficient better. This information will be kept in the fusion decision map. The fusion decision map has the same size as the original image. Each value is the index of the source image which may be more informative on the corresponding wavelet coefficient. Thus, we will actually make decision on each coefficient. There are two frequently used methods in the previous research. In order to make the decision on one of the coefficients of the fused image, one way is to consider the corresponding coefficients in the source images as illustrated by the red pixels. This is called pixel-based fusion rule. The other way is to consider not only the corresponding coefficients, but also their close neighbors, say a 3x3 or 5x5 windows, as illustrated by the blue and shadowing pixels. This is called window-based fusion rules. This method considered the fact that there usually has high correlation among neighboring pixels.
In our research, we think objects carry the information of interest, each pixel or a small neighboring pixels are just one part of an object. Thus, we proposed a region-based fusion scheme. When make the decision on each coefficient, we consider not only the corresponding coefficients and their closing neighborhood, but also the regions the coefficients are in. We think the regions represent the objects of interest. We will provide more details of the scheme in the following.
We first apply Canny edge detection on the LL band of the wavelet coefficient maps of the source images. The results are the edge images which provide the location and intensity of edges in the source images. Next, we perform region segmentation using the edge information. The output are region images, in which different values represent different regions. Then, the activity levels of each region is obtained by averaging the high-frequency wavelet coefficients, which may be more informative. Thus, we generate the region activity tables. The larger activity value mean more informative of the region. Base on the edge and region image and the region activity table, we apply the following fusion rules to compute the fusion decision map.
| 3.2 Image quality and fusion evaluation |
|---|
Assessing image fusion performance in a real application is a complicated issue. In particular, objective quality measures of image fusion have not received much attention. Some techniques to blindly estimate image quality are proposed in our research. Such quality measures can be used to guide the fusion and improve the fusion performance.
3.2.1 Histogram modeling
Here is the image model we used:
The observed image equal the signal plus the noise. Here, we are not going to study the image itself, but its edge intensity distributions. Since the edge intensity is come from the gradient information of the image, we consider using a mixture of Gaussian density to approximate the histogram of the gradient values:
Using the relationship between gradient and edge intensity, it's easy to see that we can use a mixture of Rayleigh density to approximate the histogram of edge intensity:
Here, the parameters and weights of mixture terms can be obtained using the EM algorithm. M is the number of terms used in the mixture density.
The EM algorithm is a recursive maximum-likelihood estimation of mixture density. Click 
to see more details.
to see some histogram modeling examples.
to see the experimental results
3.2.3 Quality estimation
A more general noise measurement Q can be defined as:
Q will decrease when i.i.d. Gaussian noise is added and will have the minimum value when the signal is also Gaussian. To compare with the normal SNR measurement, we may define a QR measurement as:
to see the experimental comparison between QR and SNR measurementsOther types of image degradation such as blurring are also considered in our work. We have pointed out earlier that the the term with largest variance parameter in the mixture model corresponds to the strong edges in the image. Since blurring will influence these strong edges significantly, we consider using the largest term to monitor the blurring. Thus, we may define IQ to estimate the overall image quality:
Here, noise will mainly influence Q and blurring will mainly influence 
.
The following Gaussian smoothing is used to simulate the blurring effect:
to see the experimental comparison between IQ and SNR measurementsA generalized IQ measurement can be defined as:
The function g1 and g2 will determine the relative importance of blurring and noise. We can adjust g1 and g2 to meet the different requirements in specific applications.
| 4. Potential applications of image fusion |
|---|
