Discussion Forums

Astigmatic Waterfall Filtering

15 replies [Last post]
Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196

I have created a new Blog entry at: -

http://setiquest.org/blog/astigmatic-filtering-improves-waterfall-signal-noise-ratio

It outlines the preliminary results that I obtained whilst investigating whether filtering a standard Waterfall matrix in different directions could allow you to significantly improve the signal to noise ratio. This experiment was relatively successful.

The purpose of the experiment was to provide comfirmation of my feeling that this approach would be useful; however it is not meant as a suggested methodology, but as a pointer to suggest that using 2D Wavelet transform noise reduction techniques may well generate some useful gain in signal to noise ratio.

Your comments would be welcome

Regards

Dave Robinson

maxs-pooper-scooper
Offline
Joined: 2010-07-28
Posts: 58
wavelet library

SciPy has a Wavelet library:

http://www.scipy.org/doc/api_docs/SciPy.signal.wavelets.html

I gotta keep pushin' python+numpy+scipy cool

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Many Thanks

Hi Max

That is exactly what I wanted - Although I am slowly getting up to speed with Python etc. I must agree with you, it is a very nice powerful combination. It will be a while before I am up to full speed with it, but am working hard with it, just about remembering to stop putting semicolons after each statementsmiley

Regards

Dave Robinson

maxs-pooper-scooper
Offline
Joined: 2010-07-28
Posts: 58
you can keep the semi-colons if you want

python will ignore them.
well actually it allows you to put multiple commands on a single line:

print; print foobar

but:

x=func();

works fine

jctarter
Offline
Joined: 2010-08-27
Posts: 17
Astigmatic filtering

I wonder what your approach would do to signals that are frequency modulated, such as the signals we so unprofessionally call squiggles?
Jill

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Thoughts on Squiggles

Hi Jill

I have been giving your enquiry some thought, I have mentioned my new Blog which outlines an interesting and very effective Wavelet denoising process, which uses the Astigmatic principle on either constant frequency signals, or those exhibiting a relatively gentle Doppler gradient, I reproduce the results here : -

Original Noisy Data

Processed Image

As can be seen, the resulting clean up is remarkably good, however on the images that you provided, the methodology would need to be modified somewhat, in as much as you could not use the Horizontal plane to estimate the noise environment with which to identify the threshold level to generate the Mask image, as your patterns have a significant horizontal component. However this is not to be taken that a Wavelet based cleanup process cannot be used. On the contrary, I think there are several alternative approaches to generate the Mask image, the first would be a single stage processing technique that generated an estimate of the signal in each direction of the planes using, for example a Wiener type filtering operation to extract the signal, and then generating the Mask by forming the .OR. image on the three directional thresholded/binarized Wiener images, which would be done at each scale. An alternative might be an iterrative approach where one estimates the image, then subtracts the estimate from the noisy image to get a better estimate of the noise, this all can be done to great effect in the Wavelet domain.

My conclusion is that Wavelet Transform techniques have a lot to offer in SETI processing, unfortunately I do not have the resources to fully explore the possibilities, however think that it would make a very nice degree level research project, and with your relationship to the very respected University, may well be able to organize such a project, and I would be delighted to help in any way I could

Regards

Dave Robinson

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Experiments on synthesized Squiggle Signals

Preliminary experiments have shown that in principle the Wavelet techniques outlined in the previous contribution seem to work extremely well on Squiggle signals, as shown below.

1) Original Signal in noise

2) Processed Signal

It is important to note that the experiment was 'quick and dirty' and mearly to indicate whether the technique could be used on this class of signal. The noise level was estimated by subtracting the Wiener filtered image from the noisy image at each of the Wavelet scales, and in each direction to obtain the Mask image. Better results could be obtained by applying an iterrative process to use this as an image estimate and reprocess the data.

As this approach has stirred up so little interest, I intend to do no further work on it, as I want to get on with learning Python, and start to process real data.

If there is any interest in this mechanism I will gladly answer any questions regarding the algorithm.

Regards

Dave Robinson

robackrman
Offline
Joined: 2010-04-15
Posts: 235
i have found this research interesting

A comment about the current demonstrative implementation: some weak signals in the image do not appear to be enhanced (see attached side-by-side image comparison).  Would you comment on that?  Would it be necessary to increase the number of images by multiples to handle each type of wavelet-based enhancement processing chosen to favor different type of signals (e.g., vertical, horizontal, ...)? 

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Missing signals

Hi Rob

Indeed there are missing signals, however I believe that this is totally due to my very crude method of estimating the background noise. All I was trying was a quick and dirty method of checking whether Wavelets could clean up Jill's squiggle signals, so I did the minimum possible to demonstrate that it was possible, without trying to hone it to pick up the very faint signals that you highlight. The crude method that I used to estimate the noise generated a discrimination level between noise and signal that was too large, and hence the low level signals got lost. I believe that undertaking another iterration, by subtracting this cleaned up image from the original image, would leave the noise field + the weak signals; and having a second go at doing the noise estimate, would pull out the weak signals, which could then be included into the intermediate Mask image; which would then allow regeneration of all the signals.

I believe that I have demonstrated that Wavelet smoothing can extract the signals from the noise; but would argue that my particular implementation is probably far from optimal. I was hoping that my posts would generate some interest from someone who's knowledge of Wavelets was greater than mine, and could proceed to develop a much more robust technique. If my back of the 'envelope calculations can show such an improvement, then imagine what doing it properly could achieve.

Thanks for responding, it is much appreciated

Regards

Dave Robinson

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Howdy Dave, Your experiment

Howdy Dave,

Your experiment definately shows a huge amount of background noise reduction, which would be very helpful for the setiQuest Explorer data (waterfall plot images), where folks stare at these images for hours on end.  Many weak / faint patterns can be quite difficult to see due to the huge amount of background noise in the images.  Any improvement in getting patterns to appear more clearly would be very helpful.

Have you tried re-running your experiment with "Squiggle" data such as Jill posted above?  Or would this not be possible since you can't filter the horizontal data?

Thanks,

-Don

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Using Wavelets to implement a non-linear Astigmatic Filter

Hi Guys

I have just put up a new Blog entry which describes my prelimininary experiments in applying Wavelets to remove the background noise in the typical Waterfall display. I think that the results are really first rate, and well exceed the results I reported in my previous Blog, where I had used a simple low pass linear filter to illustrate the concept of Astigmatic Filtering.

I have explained in the text how I think the concept can be extended to cope with 'squiggly' signals. I will attempt to give this a try sometime after Christmas.

The Blog is at

http://setiquest.org/blog/astigmatic-filtering-using-wavelets

Any comments should be posted here, and will be gratefully received

Meanwhile Happy Christmas to all SETI Questers, and hope you have a great 2012, maybe this is the year that ET will eventually phone Earth.

Regards

Dave Robinson

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Howdy Dave

Howdy Dave,

A Merry Christmas and Happy Holidays to you right back, and to everyone else.

"Phone" Earth?  I was hoping to "watch" their version of Jeopardy on TV (lol)!  Does SETI ever collect data in the TV spectrum, or would there be too much local noise?

Back on topic . . .

Thank you very much for taking time out of your busy schedule to continue your experiments!  Yes, it would be very helpful if someone experienced in programatic image manipulation would step up to the plate here, since your proof of concept seems quite remarkable.  Images 1, 2, 3, 5 and 6 are difficult to see since they are so small, but the difference in #6 from #5 looks great!  Can you post the code you are using for this latest round of experiments, so that someone would have a starting point from which to continue your work?

I'm not a Python programmer but would gladly run your code segments (if possible) on various real waterfall images where I have found faint patterns, to see what the results look like.

Thanks again for your experimental work, Dave.

-Don

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Astigmatic Fillter Code

I will gladly supply the code that I used to undertake the Wavelet denoising, however I must point out that the experiments were done using MathCad not Python. I have a copy of MathCad 14, complete with the Wavelet and Image Processing library, and I will forward the code on to anyone requiring it (else I can put it up on the Blog site if there is a call for it).

Now that my experiments have indicated the effectiveness of the methodology, I intend to construct a Python library to undertake the operation. However this will take a little bit of time, as I have other projects that have fallen behind while I followed up the Astigmatic Filter concept.

Regards

Dave Robinson

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Thank you for your reply,

Thank you for your reply, Dave.  Sorry, but I don't use MathCad.

Is the current code used by setiQuest to create the setiQuest Explorer waterfall images open source?  If so, what is the URL where it's located?

Thanks again,

-Don

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
Seti and MathCad

Although MathCad is an excellent tool for investigating the viability of algorithms, it really isn't suitable for production line processing of very large data sets, which is what I guess you were hoping to use it for. The way that it handles storage is that it appears to associate memory with the region of the page that it was written, and not with the sequence it was calculated. It works really well as a mathematical scratch pad - the purpose for what it was designed. However if you continuously use it it will eventually give you an 'out of memory exception', and the only way that I have found to clear it is to turn off MathCad and restart it. Bearing in mind it is a 32 bit programme when dealing with Gigabyte arrays, this happens very rapidly. This is one of the major reasons that my work has tended to use a synthetic Waterfall, rather than use a real matrix. In this respects Matlab is far better, as the user has the ability to control the memory contents, allowing used but redundant arrays to be cleared out and the memory reallocated. Unfortunately I lost access to this when I retired. I had hoped that Octave would make a good alternative, but I am afraid I was sadly disappointed with its performance.

Regards

Dave Robinson

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Hi Dave

Hi Dave,

Actually, I was asking if the code that is currently in use by setiQuest (not your experimental code) to generate the existing setiQuest Explorer website waterfall images, is available as open source or not, for personal experimentation, enhancement, etc.?

Thanks again for your help smiley!

-Don