Discussion Forums

waterfall dots

39 replies [Last post]
herpy
Offline
Joined: 2010-08-27
Posts: 65

Are dots interesting?

Here are some from the same data set which seem weird to me.

http://imageshack.us/photo/my-images/214/dots2.png/

The two pictures are about 375000 hz apart but the dots show up at around the same time.  What causes dots.  I looked to see if I could find something similar with google searches and to my surprise it seems like someone else thought dots were worthy of mention.  

The blue waterfall plot on this page has a similar looking dot.
http://argus.naapo.org/~rchilders/

 

Dave Robinson
Dave Robinson's picture
Offline
Joined: 2010-04-29
Posts: 196
The cause of dots

Dots on the Waterfall are caused by having a single tone signal that lasts for one complete line of data, but no more. The interesting thing is your statement that you have other dots occuring at roughly the same time, if they occur at EXACTLY the same time, i.e. they are on the same line number, this could indicate there may be more harmonics to the signal, which suggests that it may not be purely sinusoidal, and from their position one could do an analysis that could regenerate the overall shape of the signal.

The dot of the amplitude you show is very unlikely to be caused by noise, and thus must be a signal that lasts for some time for it to form the compact form in the Fourier transform domain; however the fact that it wasn't there on the line before, nor on the subsequent line, means that it was only present during that single line of data acquisition. This rather suggests that the signal was generated within the ATA apparatus and not an external signal, as the chances of something lasting for exactly one line and synchronized to the data acquisition system on the ATA is very remote indeed. Anyhow thats my take on the subject, maybe Sigblips may have some interesting points to make

Regards

Dave Robinson

herpy
Offline
Joined: 2010-08-27
Posts: 65
The line number for each dot

The line number for each dot is
323
362
365

I zoomed in but they don't overlap.
http://imgur.com/ihdhP

It is really close though.  To me it looks like it is the same thing just switched 375000 hz if that is even possible.

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
I see 3 pulses and a strong

I see 3 pulses and a strong constant tone. There is RFI like this all over the place in the setiQuest data. I can't say much more without any additional information. What dataset are these pulses from?

herpy
Offline
Joined: 2010-08-27
Posts: 65
2010-11-05-BD114586-1420_1-8b

2010-11-05-BD114586-1420_1-8bit

the first picture starts at around 1422658934hz
the other is around  1423033934hz

herpy
Offline
Joined: 2010-08-27
Posts: 65
oops sorry I am bad at making

oops sorry I am bad at making info graphics. I was sitting here trying to figure out what could possibly be a strong constant tone.  Then realized that the white line separating the two pictures looks like a strong constant tone.   Or am I wrong and there is some hidden strong constant tone around the dots?  I will try to be more clear next time.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Those dots are interesting. 

Those dots are interesting.  The strongest dot spans two-to-three pixels in time and frequency (see zoomed image).  What is the resolution of this image?  Approx. one-second by one-Hertz per pixel?  If so, this event represents about 1/2 arcmin rotation on the sky which is significantly smaller than the synthesized beam.  Therefore, the "dot," if from the sky, would be a time and frequency locallized pulse.  This dot  is interesting because it "appears" to be stronger than what would be likely to occur by chance, howerver, a useful project would be to determine the noise power distribution from the data and then calculate the probability of a "dot" of this power occurring in a given image.

herpy
Offline
Joined: 2010-08-27
Posts: 65
Yes it is about one second by

Yes it is about one second by one hz per pixel 

robackrman
Offline
Joined: 2010-04-15
Posts: 235
time axis

How does time flow in these images?  From top-to-bottom or bottom-to-top?  In other words, are the "dots" in the first half of the time-series data or the second half of the data?

herpy
Offline
Joined: 2010-08-27
Posts: 65
same way as all the other

same way as all the other ones.  I don't know how to show it any other way.
Like this one.
http://184.73.186.167/robs_dokuwiki/2010-11-06-epsilon-eridani-1420mhz/2010-11-06-dorothy_epsiloneridani_1420_1-chan-1552.5-palette.png

herpy
Offline
Joined: 2010-08-27
Posts: 65
Wait now I am not sure.  I

Wait now I am not sure.  I used the bmp file format to save the files and I copied various bits from differen't tutorials so I wouldn't need to include a library.  One weird thing I wasn't clear on was why you needed to save the image data reversed but I did and never really checked how the images came out so I guess it could be either way.  I will try to figure it out now.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Hi Herpy, Let me ask more

Hi Herpy,

Let me ask more directly what I am after.  I may take an independent look at the "dots," but do not want to download all 4 data files.  Are the "dots" in files 1 and 2, or files 3 and 4? My guess is files 3 and 4.

Thank you,
Rob

herpy
Offline
Joined: 2010-08-27
Posts: 65
It looks like it is reversed

It looks like it is reversed top to bottom.

http://imgur.com/bNDNm

http://setiquest.sigblips.com/analysis/2010-01-22-kepler-exo4/waterfalls.php?chan=-695

here are the bin numbers incase the frequency is off.

7028000
and
7403000

I am still not 100% sure but it looks like you need file 1 and 2.

herpy
Offline
Joined: 2010-08-27
Posts: 65
I am so confused now.  I wont

I am so confused now.  I wont be sure until I get a chance to run another data set. Even then I probably wont have much of a clue.  I thought everything worked top(oldest) to bottom(newest)  but it seems like the setiquest waterfalls are the other way.  Is there a right way to do this?  So you probably need 3 and 4.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
don't worry, i have been confused by this too

The images are refered to as "waterfalls."  If you think of water flowing over a dam, the newest water (data) is at the top and the oldest water (data) has already cascaded to the bottom.  When reading a file sequentially, the newest data is at the end (just being read) and the oldest data was at the beginning (the first bytes read).

Happy New Year!

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
Hello Rob, You got the

Hello Rob,

You got the direction on the first part of that waterfall description backwards!  (:

The two pulses you're looking for are about 1.5 seconds in duration with a 1.5 second spacing between them.  The duration and the spacing appear to be related. They are in the 2010-11-05-BD114586-1420_1-8bit-04.dat file.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
I have not yet independently

I have not yet independently confirmed the "Herpy Dots."  The most likely reason is that I downloaded the wrong file (3rd file of 4).  I will download another file from the same observation this evening.  I generated a spectrum which I will share because it has a beautiful hydrogen profile.  There is a strong tone at the bottom of the right-shoulder.

herpy
Offline
Joined: 2010-08-27
Posts: 65
that strong signal on the

that strong signal on the right is in a few other data sets at the same freq.  I tried to make a picture of it awhile ago.
http://imageshack.us/photo/my-images/8/testerw.png/

the three plots stacked on the right are lined up on the same freq. 

The slant signal was just one that almost lined up between data sets.  I guess it kept on moving between collections.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Thank you for the interesting

Thank you for the interesting images.  The tone is most likely something local.  There is no drift.  It is interesting how the tone power level varies significantly.

herpy
Offline
Joined: 2010-08-27
Posts: 65
I am beginning to think they

I am beginning to think they add it in on purpose for some reason.

http://imgur.com/YROk1

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Herpy "Dots" seen in independent analysis

The Herpy "Dots" have been verified in independent analysis (see below).  In the next step we will fit the power distribution, of which square root should be Raleigh (or more generally Chi 1), to calculate the significance of these dots appearing in this observation.  If we find that it is unlikely these dots would appear, we may choose subsequently to search for them in observations of other targets (and, I suppose, hope not to find them).  I find these dots interesting because inadvertent RFI from terrestrial equipment is not likely, from my experience, to be compact in time and frequency.  I find these dots interesting for additional reasons which I will expand upon later depending upon how further research goes.  Herpy, do you use a window in front of your 8388608 FFT?

herpy
Offline
Joined: 2010-08-27
Posts: 65
No window.  I don't really

No window.  I don't really know what a window does.

This is the code that runs the fft if that has anything to do with windowing.

//setup and run FFT

p = fftw_plan_dft_1d(fft_size , in, out, FFTW_FORWARD, FFTW_ESTIMATE);

fftw_execute(p);

 

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Thank you Herpy.

Thank you Herpy.

herpy
Offline
Joined: 2010-08-27
Posts: 65
where did you get the 8388608

where did you get the 8388608 number from?
I used a 8738132 point fft if that matters.

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
​​2^23 = 8388608

​​2^23 = 8388608

robackrman
Offline
Joined: 2010-04-15
Posts: 235
I made an incorrect

I made an incorrect assumption that you used an FFT length of nearest power of two.  By using length closest to sample length per second you have made the bins 1HZ, which is great!  Thank you for mentioning this because it clears up a discrepancy between our results: I had found the "dots" in different bins than you had specified.

herpy
Offline
Joined: 2010-08-27
Posts: 65
Which data sets would you

Which data sets would you look in to try to find more dots to prove its nothing?

I am pretty sure there are no dots in any of these data sets.

2010-11-05-zetaoph-1420_1-8bit

2010-01-22-kepler-exo4-1420mhz

2010-06-04-rosetta-8bit

2010-11-05-HD060848-1420_1-8bit

2010-11-05-HD172175-1420_1-8bit

2010-11-05-moon-1420_1-8bit

One thing I noticed is the kepler data set says its 1420 mhz but it is really 1419446400hz
which screwed me up for awhile.  Is there any way to confirm the true center freq on data sets that don't have a waterfall generated?  

herpy
Offline
Joined: 2010-08-27
Posts: 65
Here is why I want the true

Here is why I want the true center freq.
I made a wxruby script to look at all the signals I find.  You can scroll around and double click the green squares which then shows you the picture of the signal.  If the center freq is off then it wont line up right between data sets.  

Here is a screenshot of it looking at an area with alot of signals.   
http://i.imgur.com/VTDcA.png

Seems like most times there is a signal it shows up in other data sets too. But there are a few that only show up in one.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Here is a screenshot of it

Here is a screenshot of it looking at an area with alot of signals.   
http://i.imgur.com/VTDcA.png

I like that!  Would you describe the algorithm that decides if a signal is present?  Did that algorithm detect the "dots?"

herpy
Offline
Joined: 2010-08-27
Posts: 65
Unfortunately, I used my

Unfortunately, I used my eyes.

I went through all 8738 images which are 1000 pixels wide each pixel being 1hz for each data set and saved all the ones that had something in it that caught my eye.  So I can say that I am pretty sure there are no other dots but not 100% sure because I went through at 5 pictures per second.  Even looking at 5 per second my eyes work better than any algorithm I can write.

The program just highlights the box representing the picture which has the signal I saw and lines it up against all the other data sets I went through.  The white boxes are pictures I didn't see any signals in.  

 

herpy
Offline
Joined: 2010-08-27
Posts: 65
Everywhere I upload stuff

Everywhere I upload stuff seems to get taken down so where is a good place to upload a 60 megabyte zip file?

robackrman
Offline
Joined: 2010-04-15
Posts: 235
Accurate parameters for

Accurate parameters for setiQuest observations should be found in the headers.  See (http://setiquest.org/wiki/index.php/SetiQuest_Data) for a description of the header format and examples in Perl of dumping various header fields.

You are performing some development in Ruby, so here is an example in Ruby (under irb) of downloading the first header and printing the observation center frequency of the Kepler observation mentioned:

rackrman@dev:~$ irb
irb(main):001:0> require 'net/http'
=> true
irb(main):002:0> http = Net::HTTP.new("setiquest.sigblips.com")
=> #<Net::HTTP setiquest.sigblips.com:80 open=false>
irb(main):003:0> req = Net::HTTP::Get.new("/download/2010-05-14-kepler04-3/2010-05-14-kepler04-3-hdrs.dat")
=> #<Net::HTTP::Get GET>
irb(main):004:0> req.set_range(0, 80) # get first header, not entire file
=> 0...80
irb(main):005:0> rspns = http.request(req)
=> #<Net::HTTPPartialContent 206 Partial Content readbody=true>
irb(main):006:0> p rspns.body.unpack("x24d1")
[1413.4464]
=> [1413.4464]

Note: the center frequency field contains the value 1413.4464.  This appears to be low based on your comparisons.  I think there is an offset involved (channel next to DC channel) but don't remember.  I no longer work at the SETI Institute - have not been involved in observations for months.  Perhaps, one of the current observers can comment on the meaning of the center frequency header field.

herpy
Offline
Joined: 2010-08-27
Posts: 65
as far as I can tell it

as far as I can tell it always returns 1413.4464 no matter what hdrs file you use it on.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
That is disappointing.  If at

That is disappointing.  If at some point the setiQuest forum is graced with the presence of a SETI Institute staff, perhaps they can explain the most reliable way to determine the observation frequency.

herpy
Offline
Joined: 2010-08-27
Posts: 65
Would using the hydrogen peak

Would using the hydrogen peak as a guide even be worth trying to figure out?
wikipedia has the hydrogen freq as 1420.40575177
but this picture makes it seem like something is off :
http://setiquest.sigblips.com/analysis/2010-03-19-exo4/

and if you look at these two it doesn't line up

http://setiquest.sigblips.com/analysis/2010-11-05-dorothy_exo-gl581-1420_1/

http://setiquest.sigblips.com/analysis/2010-11-05-kepler04-1420_1/

robackrman
Offline
Joined: 2010-04-15
Posts: 235
In the direction of

In the direction of observation there are clouds of hydrogen gas in the Milky Way that are moving at different velocities with respect to Earth which causes the hydrogen line to broaden and the "peak" to shift due to Doppler, therefore, the hydrogen profile is not useful for exactly determining the frequency of observation.  Observations that contain the hydrogen profile in most cases were observed at 1420.0 MHz.  In the general case, the hydrogen profile may not be present in the spectrum because the frequency of observation for the current ATA could be anywhere in the range 0.5 to 11.5 GHz.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
If we are proposing that the

If we are proposing that the "dots" are sent by ET (or some other unknown distant unique natural celestial source), then we might expect not to see them in observations in other directions on the sky.  It is encouraging that the six other sources you have examined do not contain "dots."  If our investigation shows the power of the "dots" to be significant (not likely to occur multiple times in an observation) then there might be justification to reobserve source BD114586 for a longer period to see if the dots appear again (are not, for example, some overpassing RFI).

robackrman
Offline
Joined: 2010-04-15
Posts: 235
The attached 3D plot

The attached 3D plot illustrates the significance of the strongest "Herpy Dot" towering over the noise floor.  It appears, some non-random event energized a ~ 1Hz Fourier bin for about one second.  Very interesting that there were three of these events in one observation.  I may go on at some point and fit a distribution to the noise floor to calculate the probability of an event of this magnitude or greater.  Since Herpy first found these "dots" and now, I have lost motivation, probably because there is no hint that anything we do here in the forum would have any influence on future telescope activities.

robackrman
Offline
Joined: 2010-04-15
Posts: 235
update

Herpy Dots have been found in SETILive data.  These dots, which are localized in time and frequency, are not random Gaussian tail events.  That is evident from their number, pattern, and strength in the image.  Note that they are much stronger in one beam than the other which suggests non-local origin.  Very interesting.  I wonder if someone who can, will investigate these curious anomalies...

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
It was a lot of work but I

It was a lot of work but I hunted around and found the GSL_ reference number for your SETI Live image:

http://talk.setilive.org/observation_groups/GSL00040m7

Rob, in the future it would be greatly appreciated if you could include any relevant ID numbers or links for images that you post. It will make it a whole lot easier tracking it back.  

These pulse patterns are not that uncommon in data from the second SETI Live tuning.  They are part of a wider structure and they have a number of interesting time and frequency relationships.  I have no idea what this signal is but it is fairly strong and it occurs in multiple beams which says RFI to me.  But you bring up an important point that the signal is much weaker in the second beam.  How much weaker should it be if it's not sidelobe leakage? Good question.  I don't know, I'm not sure how well this is really understood.

Now here are two excellent possible candidate examples of strong individual short-duration pulses that are only in one beam. 

http://talk.setilive.org/observation_groups/GSL0001grh
http://talk.setilive.org/observation_groups/GSL0002rc5

These single beam pulse events are fairly rare.