Discussion Forums

This is weird maybe?

30 replies [Last post]
herpy
Offline
Joined: 2010-08-27
Posts: 65

Nope My program was screwed sorry.

Here is the picture if you want to laugh.  For some reason that data got repeated and triggered my script. 

http://www.shackpics.com/viewer.x?file=what_hl_1abfrc6zhds8jo0mjtc5.png
http://www.shackpics.com/viewer.x?file=what_2_cwd571ya6avwb9jz3pr9.png

herpy
Offline
Joined: 2010-08-27
Posts: 65
Lagrange-4-3991 data set

Lagrange-4-3991 data set These are the obvious signals. Is that horizontal distortion supposed to be there or am I doing something wrong still?
http://img574.imageshack.us/img574/2075/test1k.png

And what are some good examples of drifting pulse signals?

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
That looks odd. It is filled

That looks odd. It is filled with signals? At regular frequency intervals. How much bandwith does that image cover? (Or did you stitch many bands together?)

herpy
Offline
Joined: 2010-08-27
Posts: 65
I made a program to find the

I made a program to find the obvious signals and put them together. If it finds a signal it cuts out 20hz  and puts it in a picture. Then I stuck them all together. I was just wondering why they all have a similar horizontal distortion.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Very cool. Good question

Very cool. Good question about the distortion. It could be noise in the ATA.

Dispersed pulses look like this. They don't modulate other signals like the distortion you see does.

jill
Offline
Joined: 2010-01-26
Posts: 27
repetitive signals

strong signals can alias into other frequency bands because our filters aren't perfect.  note that i think there are at least two similar but distinct modulation schemes here.  a nice widget to have would be one that aligned signals at t=0 (or t=mid observation), putting signal into same channel, and then displaying the difference.  as we think about citizen scientists comparing newly detected patterns against a rogues' gallery of known RFI, i can imagine lots of uses for such comparison tools.

an analysis of the differences in frequency between these multiple signals might also prove instructive.  take the one that has the largest total power when summed over the observation and assume that's the fundamental or real signal - weaker ones should be aliases, and the farther from the fundamental the weaker they should be.

jill

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Thank you for those

[no-glossary]
Thank you for those ideas.

The dynamic range of the waterfall plots is 24 dB.  The nominal rejection of the setiQuest WOLA-FFT channels is greater than 40dB.  My first reaction is that a very strong signal clipped to the highest pixel value would be required to see its alias in a neighbor channel.  But, I have not yet thought it through carefully.

I plan to synthesize some test signals in noise and then pass them through the setiQuest processing to see what happens and will report results here in the forum.
[/no-glossary]

gerryharp
Offline
Joined: 2010-05-15
Posts: 365
autoscaling

Just to reinforce Jill's point about autoscaling before striping the data together. If the original waterfalls are autoscaled, then the aliases may look just as bright as the true signal.

Responding to Mike's (out of band) point, if Rob see's the same signals as herpy, then it is probably there, in the data.

Very cool!

Gerry

mikedavis
Offline
Joined: 2010-04-16
Posts: 3
LO Instability?

As I said, I'm just getting used to this messaging system. My reply to Gerry and the rest of you should have appeared here, not at the top of this thread. Sorry about that.

Mike

robackrman
Offline
Joined: 2010-04-15
Posts: 235
I think Mike is suggesting

I think Mike is suggesting that the signals might be independent narrow carriers that are affected together by an oscillator instability somewhere in the signal chain.  So, if I understand his theory, we should attempt to refute it by searching for a signal in the data that does not "wiggle" in the same manner.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Here are three (of many)

[no-glossary]
Here are three (of many) waterfall plots from a 2010-05-07 C-Band galactic anticenter observation that appears to show copies of the same signal:

Here is the URL for the entire set of waterfalls for this observation: http://184.73.186.167/analysis/2010-05-07-galanticenter-2/
[/no-glossary]

herpy
Offline
Joined: 2010-08-27
Posts: 65
Only used the first data file

Only used the first data file for the galanticenter plot.
Why does the spacing matter?

http://www.shackpics.com/viewer.x?file=test_test_aso643egxo9fmj6mn5nn.png

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Hi Herpy, To make sure I (and

[no-glossary]
Hi Herpy,
To make sure I (and other readers) understand your recent plot post (http://www.shackpics.com/viewer.x?file=test_test_aso643egxo9fmj6mn5nn.png):  it shows a zoom of ~ 1670 Hz bandwidth - Lagrange 4 on top and galactic anticenter (thinner) on the bottom - demonstrating that the signal spacing is the same for two separate observations on different days: ~ 1267 Hz.  Is that correct?

The spacing value is relevant (a clue), because if there is a problem in the ATA signal chain, it will have some numerical relationship with oscillator frequencies, sampling rate, FFT size, and other values in the system and possibly their combinations.

Nice work and very helpful.
Thank you,
Rob
[/no-glossary]

herpy
Offline
Joined: 2010-08-27
Posts: 65
Yeah, I am pretty sure each

Yeah, I am pretty sure each pixel is 1 Hz.

herpy
Offline
Joined: 2010-08-27
Posts: 65
Here is one showing the

Here is one showing the offset if that matters. I lined them up on the same bin.

http://www.shackpics.com/viewer.x?file=offset_fi20yvpwkcz1200pvoiv.png

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Just to reinforce Jill's

[no-glossary]
Just to reinforce Jill's point about autoscaling before striping the data together. If the original waterfalls are autoscaled, then the aliases may look just as bright as the true signal.

The signal can sometimes be found twice in the same FFT sub-channel, for example: http://184.73.186.167/analysis/2010-10-08-Lagrange-4_3991_3/waterfalls.p...
[/no-glossary]

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
A lot of DSP "amateurs" on

A lot of DSP "amateurs" on this forum (myself included) has been asking how to render waterfall plots. If you have got it work, is there any chance you can share your code?

herpy
Offline
Joined: 2010-08-27
Posts: 65
I am not sure its working, it

I am not sure its working, it seems to be working. It is just a small c++ program that runs a 8million point fft on the data and spits it back out 4 times bigger. It turned the 2010-10-01-exo-gl581_2-8bit data set into a 56 gig file. Then I made a ruby script to draw whatever range of bins you want to look at. I will post it on gist in the morning unless there is a better way.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Ah, so you succeeded in

Ah, so you succeeded in "zooming" by increasing FFT size? And then you FFT successive chunks of the input data and render them underneath each other?

herpy
Offline
Joined: 2010-08-27
Posts: 65
Yeah that sounds about

Yeah that sounds about right.  Instead of rendering it I write it to a file as float,float complex pairs so I can do the normalization depending on how much you are looking at. I am not sure I succeeded but it seems like it is working. Not sure if I am using github right.

https://gist.github.com/62d0900692195fac869e

I am not really good at programming and the ruby parts are really slow but I used ruby inline for some of the really slow parts and that helped a lot.

So if you want to use it you have to set the #define INPUT_FILE to the data you downloaded from the website and set the #define OUTPUT_FILE to whatever you want the 4x bigger converted file to be called. You run the program after you complie it and wait till it finishes. Then you set the file_path in the ruby script to the output file and it should all work from there.

I used Ubuntu 10.10 32 bit. I used codeblocks for the C++
You need the FFTW library for c++ but I think Ubuntu has it already not sure though. 
And if you use the ruby script You need to get RMagick working.

herpy
Offline
Joined: 2010-08-27
Posts: 65
oops I forgot to add the

oops I forgot to add the CombTheDesert.rb file. It is in there now. 

And I used ruby version 1.9 not sure if that matters.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Thanks!

Thanks!

robackrman
Offline
Joined: 2010-04-15
Posts: 235
That is a fascinating plot. 

[no-glossary]
That is a fascinating plot.  Many signals are expected in C-Band from communications satellites; however, your plot seems to show, from looking at the "wavy" part ("horizontal distortion") of the vertical signals, what looks like copies of a particular signal.  I don't know what to make of that at the moment.  I will pass the same data through the setiQuest waterfall generation code and then we can compare notes.  Please verify that the data from which your plot is derived is from files: 2010-10-08-Lagrange-4_3991_3-8bit-{01,02,03}.dat  I included a crop and contrast enhancement of your plot below for other readers to conveniently reference.
[/no-glossary]

herpy
Offline
Joined: 2010-08-27
Posts: 65
Yes it is from

Yes it is from lagrange-4-3991. If it isn't normal then I am probably doing something wrong. It is just that I noticed that all the signals looked the same so I put them together to get a better look.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
We have similar results!  The

[no-glossary]
We have similar results!  The "horizontal distortion" is in the data, not something to do with your analysis.  For a few examples (from many in the setiQuest analysis waterfalls) see:

http://184.73.186.167/analysis/2010-10-08-Lagrange-4_3991_3/waterfalls.p...
http://184.73.186.167/analysis/2010-10-08-Lagrange-4_3991_3/waterfalls.p...
http://184.73.186.167/analysis/2010-10-08-Lagrange-4_3991_3/waterfalls.p...
http://184.73.186.167/analysis/2010-10-08-Lagrange-4_3991_3/waterfalls.p...
http://184.73.186.167/analysis/2010-10-08-Lagrange-4_3991_3/waterfalls.p...

This does not "feel" right.  Do these similar signals really exist at their respective frequencies, or is the system somehow producing images (copies) in the data?  Further investigation is warranted.
[/no-glossary]

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
Repetitive copies of signals

Repetitive copies of signals have been seen in many of the other setiQuest data sets. Not to the extent of the thousands of signals found here but 3991 MHz was said to be a "bad band."  So was this just some C-band satellite that got in the way or was it distortion caused by the ATA?

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Hi "sigblips," It would be

Hi "sigblips,"
It would be extremely helpful if you would run the Lagrange 4 (3991 MHz) data through baudline and then compare your results to Herpy and setiQuest results.
As Jill suggested, the distance between signal "copies" in frequency may shed some light on what is happening (e.g., alisasing, mixing, ...)
Thank you,
Rob

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
I see thousands of evenly

I see thousands of evenly spaced duplicate signals. I'm not done with my analysis yet and there are a whole lot of TBD's that need to be filled in but here is a preliminary work-in-progress post:

http://baudline.blogspot.com/2010/11/setiquest-lagrange-4.html

The signal shape and spacing matches what Rob sees.  The exact modulation scheme and significance of the ∆1267.226 Hz spacing is TBD but I already know it will be interesting.

gerryharp
Offline
Joined: 2010-05-15
Posts: 365
not necessarily uncommon

Hi Sigblips

I congratulate you on discovering a very rare type of wide-band signal encoding. I have seen this behavior only once before in my life. Rob may remember that back in teh (Rapid Prototype Array) days, 8 years ago, we were studying the output of some L-band satellites. Rob wrote a real-time spectrum analyzer with a zoom button, and we discovered a comb of signals similar to the ones you describe. In this case,each tine of the comb moved up and down asynchronously (that is, amplitude modulation). We speculated that each of these tones might be carrying a narrow band telephone signal and it would be funny to decode them!

In your case the signals appear to move (side to side?) in unison. Can you verify my understanding? If this is the case, the information on each vertical line is the same. THere is a good reason to do this -- to overcome jamming by radio frequency interference. For example, by averaging the signals from all lines, you improve (decrease) the error rate. Since they are all sine waves, you can easily estimate and remove the dispersion in the ionosphere for each signal individually and they line up perfectly. Finally if RFI covers some of the lines but not all, there are simple algorithms to reject "outlier" data and still recover your message.

Everything I've said would be true both for an ET signal and a man-made signal. Since we're in a "bad band", the chances of human-made is higher than the already very high probablity in clean bands.

Also, not to leave out herpy and Rob, the same discussion would be true for any comb of lines.

Still, this is a very interesting discovery. Can you line up (by shifting the plots in freqeuncy) and then correlate one line with all the others? That is, you get one correlation coefficient for each pair of lines. From this you can produce a quantitative measure of how similar lines are.

If you discover the lines are very similar (probably yes), then how about lining up all the plots in frequency and then suming them? This would give you a much higher SNR representation of the information encoded in the signal. Can you learn more about what kind of infomration the signal contains by seeing it with high SNR?

This is very cool and I encourage anyone interested to try to help us understand the character of this signal even more.

Thanks and nice work!

Gerry

mikedavis
Offline
Joined: 2010-04-16
Posts: 3
LO Instability?

Hi Gerry et al. -
Apologies for 'out of band' response - I'm just getting used to the SetiQuest messaging system. I agree, after reading past Herpy's post last Sunday that the effect is real.

I can see an easy explanation for this type of behaviour. I suspect that a single local oscillator is used to control the frequency of all of these signals. In that case, if the oscillator is unstable (for whatever reason) then all of the signals will show the -same- instability. Oscillator instability at the few-Hz level can arise from any number of reasons, and is almost certain to occur unless the oscillator is controlled by an atomic standard or very carefully temperature stabilized crystal. Thoughts on this???

herpy
Offline
Joined: 2010-08-27
Posts: 65
http://img709.imageshack.us/i

http://img709.imageshack.us/img709/3716/thincut.png

long horizontal streaks with what looks like shadows to the right.
Is this some kind of obvious RFI?