This document outlines some interesting results following a simple experiment I have undertaken in preparation for undertaking a much more elaborate experiment into applying the Donoho & Johnstone methodology for noise reduction using a 2D Wavelet decomposition of a Waterfall plot. It should be emphasized that the results shown here are NOT based on this technique, but are based purely on linear filtering, and were meant to be a preliminary test of the validity of my assumptions.
The Waterfall plot has some interesting properties, in as much as the detail along the frequency (horizontal) axis is contained mainly in the fine detail. Typically the width of a signal can be as narrow as a single sample. However in the Time (vertical) direction is contained mainly in the very course detail. For example consider the rather uninteresting case of a fixed tone, in the time direction will have a width of the complete height of the Waterfall plot. Angled traces will still contain quite wide tracks in the time direction.
A 2D Wavelet transform decomposes a matrix into sub-matrices of detail scale, and position in the horizontal, vertical and diagonal directions. Donoho & Johnstone have shown that by shrinking the Wavelet coefficients in the scales that mainly contain noise allows a remarkable non-linear noise reduction; and what is more – on the 2D transform, we can apply this approach in the different directions of the matrix. Thus although we can't do much in the horizontal plane, which would damage the integrity of the reconstruction, as it would smear the one sample signal points; there is no problem in removing any fine detail in the vertical direction, as we know that this can only be down to noise. Thus I believe that this technique will allow us to do a great deal towards significantly improving the signal to noise ratio of the standard Waterfall plot.
Unfortunately I currently don't have access to any Wavelet libraries with which to try to see if this concept actually works in practice. (Does anyone know if there is a Python based library? I would be grateful for a pointer – my own Python skills have yet to reach the level where I could consider writing my own). However whilst thinking about how to convince myself that such an approach was worth putting a lot of effort into, I began to think of a more mundane experiment I could try and see if the concept of filtering differently in the two directions actually worked (I call this approach 'astigmatic filtering' as we are blurring the matrix differently in the horizontal and vertical plane.
I used my synthetic Waterfall matrix that I used for my Radon Transform experiments, as I knew what the data looked like prior to it getting polluted with noise.
I then decided to check the Fourier Transforms of the Matrix in both the horizontal direction (which of course showed that you needed the full bandwidth to be able to reconstruct the matrix as the data contained 1 sample wide information ) and the vertical direction. The result is shown in the following diagram ( the red plot)
This graph was generated by Fourier transforming each column of the Waterfall matrix, then summing across all of the columns, then calculating the amplitude of the complex numbers. Notice how small the overall span of 'frequency' space is used in the vertical direction. This suggests that it would be perfectly safe to low pass filter the vertical direction of the Waterfall matrix – providing we don't apply any filtering to the horizontal direction. As this was a quick experiment to give me the 'warm and fuzzies' regarding the concept of astigmatic filtering, I simply generated the low-pass filter shown in blue – it hasn't been optimized in any way shape or form, and is a simple Gaussian curve.
Processing consisted of taking an individual column of the Waterfall plot Fourier transforming it, multiplying by the Gaussian filter, then inverse transforming (i.e fast convolution). I first applied this to the unpolluted Synthetic Waterfall matrix and found that the filtered result was visually identical to the original (MSE was approximately 10-5). So I was relatively happy that I wasn't actually damaging the original data. I then added in a fairly heavy chunk of noise to the Waterfall Matrix. I show this as a surface plot, as I believe it is more informative than an image.
I then repeated the astigmatic filtering operation, which generated the following results.
The purpose of publishing this document is not to recommend this linear astigmatic filter technique as a practical method to be used, but hopefully to act as a catalyst to someone who has access to a Wavelet library to try the Donoho & Johnstone methodology which I am now sure should produce some really useful results; as it may be sometime before I get access to the code that will allow me to implement it.
Comments are welcome. Join the conversation at http://setiquest.org/forum/topic/astigmatic-waterfall-filtering