This document describes some preliminary results obtained by using a Wavelet based noise removal technique to implement the astigmatic filtering briefly outlined in a previous blog. As before, the same synthetic Waterfall plot has been used to evaluate, and demonstrate the effectiveness of the methodology, which appears to be very good at removing the typical background noise present in the Waterfall diagrams. However note that unlike the Radon transform methodology which provides the capability to dig out signals well below the ambient noise level, the Wavelet technique will only clean up signals for which there is statistical evidence for their existence, (e.g. they must be above the ambient noise level).
The previous work illustrated that the ambient noise level could be significantly reduced by noting that the frequency domain characteristics were significantly different in the time dimension (vertical direction) and the frequency dimension (horizontal direction), and further showed that a significant increase in the signal to noise ratio could be obtained by applying a linear filter along the time axis, but not the frequency axis. However as was stated, the purpose of the document was to encourage someone to investigate the use of 2D Wavelets to apply the Donohoe and Johnston noise reduction technique on the data. However with no response to my challenge, I have followed up the work which is what this document briefly describes.
Treating the Waterfall data as a straight forward 2D matrix, one can apply a 2D Wavelet transform to it. This will decompose the matrix into a set of levels, with each level having half the resolution of the preceding stage; in addition each level consists of three sub-matrices, one representing the horizontal detail, one representing the diagonal detail, and one representing the vertical detail. Observation of a typical level of a transformed Waterfall matrix shows one interesting characteristic. This being that virtually all of the information, is in the vertical direction. The other 2 directions for all practical purposes hold noise.
1) Horizontal Detail
2) Diagonal Detail
3) Vertical Detail
The approach that was adopted for this experiment was to use the Horizontal direction matrix to make an estimate of the ambient noise level; this value was then used to determine a statistically meaningful threshold that could be applied to the Vertical direction matrix, in order to generate a binary mask that was zero at positions where the values that were indistinguishable from noise, and set to one where we could say with relative certainty that there was a signal. This process was undertaken for each scale level of the Wavelet transform. A typical mask image is shown below.
4) Mask Image
Although there is a small residual quantity of noise left, it can be seen that this Mask isolates pixels within the matrix that contains signal, at the scale setting corresponding to the specified Wavelet level. The next task is to use these masks on all the three directional planes (e.g. the horizontal, diagonal and the vertical plane. This can be done simply by doing a pixel by pixel multiplication of the detail planes by the corresponding mask. Note what this means, we have created a synthetic Wavelet transform in which the pixels corresponding to geographic areas known to have signals are set to the original coefficients, but pixels that do not correspond to signals are clamped to zero. The next step is to simply undertake an inverse Wavelet transform to generate the following result.
As can be seen the result is an outstanding improvement; the Wavelet based non-linear filter provides a far more effective noise removal technique than the linear filter shown in my previous Blog, and appears to confirm my belief that the concept of Astigmatic filtering to treating the Waterfall matrices could well show great benefit. Note that for 'squiggly signals, the concept can be extended to create a mask image which uses a combination of both the Vertical and the Diagonal components at each scale.
5) Original Noisy Synthetic Waterfall Image
6) Above image after processing