I have created a new Blog entry at: -
It outlines the preliminary results that I obtained whilst investigating whether filtering a standard Waterfall matrix in different directions could allow you to significantly improve the signal to noise ratio. This experiment was relatively successful.
The purpose of the experiment was to provide comfirmation of my feeling that this approach would be useful; however it is not meant as a suggested methodology, but as a pointer to suggest that using 2D Wavelet transform noise reduction techniques may well generate some useful gain in signal to noise ratio.
Your comments would be welcome
SciPy has a Wavelet library:
I gotta keep pushin' python+numpy+scipy
That is exactly what I wanted - Although I am slowly getting up to speed with Python etc. I must agree with you, it is a very nice powerful combination. It will be a while before I am up to full speed with it, but am working hard with it, just about remembering to stop putting semicolons after each statement
python will ignore them.
well actually it allows you to put multiple commands on a single line:
print; print foobar
I wonder what your approach would do to signals that are frequency modulated, such as the signals we so unprofessionally call squiggles?
I have been giving your enquiry some thought, I have mentioned my new Blog which outlines an interesting and very effective Wavelet denoising process, which uses the Astigmatic principle on either constant frequency signals, or those exhibiting a relatively gentle Doppler gradient, I reproduce the results here : -
Original Noisy Data
As can be seen, the resulting clean up is remarkably good, however on the images that you provided, the methodology would need to be modified somewhat, in as much as you could not use the Horizontal plane to estimate the noise environment with which to identify the threshold level to generate the Mask image, as your patterns have a significant horizontal component. However this is not to be taken that a Wavelet based cleanup process cannot be used. On the contrary, I think there are several alternative approaches to generate the Mask image, the first would be a single stage processing technique that generated an estimate of the signal in each direction of the planes using, for example a Wiener type filtering operation to extract the signal, and then generating the Mask by forming the .OR. image on the three directional thresholded/binarized Wiener images, which would be done at each scale. An alternative might be an iterrative approach where one estimates the image, then subtracts the estimate from the noisy image to get a better estimate of the noise, this all can be done to great effect in the Wavelet domain.
My conclusion is that Wavelet Transform techniques have a lot to offer in SETI processing, unfortunately I do not have the resources to fully explore the possibilities, however think that it would make a very nice degree level research project, and with your relationship to the very respected University, may well be able to organize such a project, and I would be delighted to help in any way I could
Preliminary experiments have shown that in principle the Wavelet techniques outlined in the previous contribution seem to work extremely well on Squiggle signals, as shown below.
1) Original Signal in noise
2) Processed Signal
It is important to note that the experiment was 'quick and dirty' and mearly to indicate whether the technique could be used on this class of signal. The noise level was estimated by subtracting the Wiener filtered image from the noisy image at each of the Wavelet scales, and in each direction to obtain the Mask image. Better results could be obtained by applying an iterrative process to use this as an image estimate and reprocess the data.
As this approach has stirred up so little interest, I intend to do no further work on it, as I want to get on with learning Python, and start to process real data.
If there is any interest in this mechanism I will gladly answer any questions regarding the algorithm.
A comment about the current demonstrative implementation: some weak signals in the image do not appear to be enhanced (see attached side-by-side image comparison). Would you comment on that? Would it be necessary to increase the number of images by multiples to handle each type of wavelet-based enhancement processing chosen to favor different type of signals (e.g., vertical, horizontal, ...)?
Indeed there are missing signals, however I believe that this is totally due to my very crude method of estimating the background noise. All I was trying was a quick and dirty method of checking whether Wavelets could clean up Jill's squiggle signals, so I did the minimum possible to demonstrate that it was possible, without trying to hone it to pick up the very faint signals that you highlight. The crude method that I used to estimate the noise generated a discrimination level between noise and signal that was too large, and hence the low level signals got lost. I believe that undertaking another iterration, by subtracting this cleaned up image from the original image, would leave the noise field + the weak signals; and having a second go at doing the noise estimate, would pull out the weak signals, which could then be included into the intermediate Mask image; which would then allow regeneration of all the signals.
I believe that I have demonstrated that Wavelet smoothing can extract the signals from the noise; but would argue that my particular implementation is probably far from optimal. I was hoping that my posts would generate some interest from someone who's knowledge of Wavelets was greater than mine, and could proceed to develop a much more robust technique. If my back of the 'envelope calculations can show such an improvement, then imagine what doing it properly could achieve.
Thanks for responding, it is much appreciated
Your experiment definately shows a huge amount of background noise reduction, which would be very helpful for the setiQuest Explorer data (waterfall plot images), where folks stare at these images for hours on end. Many weak / faint patterns can be quite difficult to see due to the huge amount of background noise in the images. Any improvement in getting patterns to appear more clearly would be very helpful.
Have you tried re-running your experiment with "Squiggle" data such as Jill posted above? Or would this not be possible since you can't filter the horizontal data?
I have just put up a new Blog entry which describes my prelimininary experiments in applying Wavelets to remove the background noise in the typical Waterfall display. I think that the results are really first rate, and well exceed the results I reported in my previous Blog, where I had used a simple low pass linear filter to illustrate the concept of Astigmatic Filtering.
I have explained in the text how I think the concept can be extended to cope with 'squiggly' signals. I will attempt to give this a try sometime after Christmas.
The Blog is at
Any comments should be posted here, and will be gratefully received
Meanwhile Happy Christmas to all SETI Questers, and hope you have a great 2012, maybe this is the year that ET will eventually phone Earth.
A Merry Christmas and Happy Holidays to you right back, and to everyone else.
"Phone" Earth? I was hoping to "watch" their version of Jeopardy on TV (lol)! Does SETI ever collect data in the TV spectrum, or would there be too much local noise?
Back on topic . . .
Thank you very much for taking time out of your busy schedule to continue your experiments! Yes, it would be very helpful if someone experienced in programatic image manipulation would step up to the plate here, since your proof of concept seems quite remarkable. Images 1, 2, 3, 5 and 6 are difficult to see since they are so small, but the difference in #6 from #5 looks great! Can you post the code you are using for this latest round of experiments, so that someone would have a starting point from which to continue your work?
I'm not a Python programmer but would gladly run your code segments (if possible) on various real waterfall images where I have found faint patterns, to see what the results look like.
Thanks again for your experimental work, Dave.
I will gladly supply the code that I used to undertake the Wavelet denoising, however I must point out that the experiments were done using MathCad not Python. I have a copy of MathCad 14, complete with the Wavelet and Image Processing library, and I will forward the code on to anyone requiring it (else I can put it up on the Blog site if there is a call for it).
Now that my experiments have indicated the effectiveness of the methodology, I intend to construct a Python library to undertake the operation. However this will take a little bit of time, as I have other projects that have fallen behind while I followed up the Astigmatic Filter concept.
Thank you for your reply, Dave. Sorry, but I don't use MathCad.
Is the current code used by setiQuest to create the setiQuest Explorer waterfall images open source? If so, what is the URL where it's located?
Although MathCad is an excellent tool for investigating the viability of algorithms, it really isn't suitable for production line processing of very large data sets, which is what I guess you were hoping to use it for. The way that it handles storage is that it appears to associate memory with the region of the page that it was written, and not with the sequence it was calculated. It works really well as a mathematical scratch pad - the purpose for what it was designed. However if you continuously use it it will eventually give you an 'out of memory exception', and the only way that I have found to clear it is to turn off MathCad and restart it. Bearing in mind it is a 32 bit programme when dealing with Gigabyte arrays, this happens very rapidly. This is one of the major reasons that my work has tended to use a synthetic Waterfall, rather than use a real matrix. In this respects Matlab is far better, as the user has the ability to control the memory contents, allowing used but redundant arrays to be cleared out and the memory reallocated. Unfortunately I lost access to this when I retired. I had hoped that Octave would make a good alternative, but I am afraid I was sadly disappointed with its performance.
Actually, I was asking if the code that is currently in use by setiQuest (not your experimental code) to generate the existing setiQuest Explorer website waterfall images, is available as open source or not, for personal experimentation, enhancement, etc.?
Thanks again for your help !