Discussion Forums

Rollout of new setiquest data and algorithms "kit"

7 replies [Last post]
gerryharp
Offline
Joined: 2010-05-15
Posts: 365

Any moment now, we'll be announcing new setiQuest data offerings and a new suite of open-source tools that you can use to analyze the data.
Here's a bit more info about the data and open-source programs. Starting late 2009, the SETI Institute began developing cutting edge of technology at the ATA to capture high speed telescope data in digital format that can be used for offline analysis. A substantially expanded archive of setiQuest data are now available to all SETI enthusiasts and hosted through a generous donation by Amazon Web Services including more than 3 TB of binary data available for download. By any measure, this is a huge quantity of data, much of which has never been analyzed before, by anyone!
SETI Institute scientists are using these data to develop new methods for signal analysis for our next generation SETI detectors. One such analysis is called "autocorrelation" and this approach is sensitive to arbitrary signals (not just sine waves) that have some kind of repeating structure (described in Ch.4 of this book). To pursue these and other analyses we have developed a suite of command-line programs (written in C) that can be linked together to do an endless variety of different kinds of analyses. Bringing these programs to the public was generously supported by Google through the Google Summer of Code program with excellent contributions from our mentee, Aditya Bhatt, an outstanding computer science undergraduate in Gandhinagar, India. Each program does just one simple operation on the data and are strung together using Linux pipes for sophisticated analyses.  Our algorithms leverage the extremely fast Fourier Transform library, FFTW, and is designed for good efficiency. Savvy users can invent their own modules to mix and match. These utilities can be used as is, or compiled into a library and called from Python, Ruby, C++, Java or your favorite programming language. We have also developed some simple prototypes of analyses in MatLab/Octave, and will be adding these prototypes to our website soon, for those who prefer this approach.
The program outputs include simple ASCII text which can be plotted in your favorite graphing program (e.g. Excel) or waterfalls plots which are a power tool for visualizing the time dependence of signals. Some waterfalls from setiQuest data are linked right on the data page, such as here. The same programs were used to produce waterfall plots for setiQuest Explorer, another citizen science program we're working on.
While we've substantially upgraded our documenation, we're still far from where we'd like to be. There are bound to be many questions from new users on how to do this or that analysis (e.g. look for amplitude-modulated signals in maser data), jow to plot the data, and so on. We're critically low on resources for this support right now, because we're focusing most of our energy on getting the ATA back up and running (this is good news!). So we're looking to build a community of support, someone to create a FAQ, someone to help users with install questions, add links to radio imagery around our source directions, etc
As we continue to break down barriers and bring SETI to the world, we hope that you will join our community and participate in the democratization of SETI science. We're still just beginning and we need your help in explaining the science of SETI to the world. We welcome your involvement with testing and developing new approaches that will reach out to people at all skill levels. Hope to see you soon!
 

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Windows Binaries?

Is anyone here able to compile/create Windows binary files (.exe format) of these command-line tools?

Thank you,

-Don

jrseti
jrseti's picture
Offline
Joined: 2010-07-22
Posts: 250
Don, We haven't tried to do a

Don,
We haven't tried to do a Windows port of these tools. But if you want to try and do that, we'll wecome the input!
 
 

-jrseti

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Would love to, but ...

Howdy jrseti,

Would love to, but I recently had to replace my PC and don't have any programming tools installed on the new box yet.  So, no promises for now.  That's why I asked if anyone was able to do this.  I don't currently have a Linux disto installed either (virtual machine under Win 7), as the projects I needed it for in the past are done.  Maybe these SETI tools will motivate me to download and install the current versions of my programming tools and a new Linux distro.

What Linux distro is favored in this group of SETI folks?

Merry Christmas,

-Don

robackrman
Offline
Joined: 2010-04-15
Posts: 235
cygwin

Alternatively, investigate Cygwin (http://www.cygwin.com) under which you should be able to compile and run the setiQuest utilities in the native Windows environment if you desire (although I have not tried to do this myself).  There is also the benefit that you will have SSH, curl, and a bunch of other handy Unix utilities available to you.  You will have to install GCC and CMake from the Cygwin repository and also probably some libraries, such as FFTW.

Rob

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Thank you for reminding me of

Thank you for reminding me of these tools, Rob.  I used them in the past when working on another space-based open source project (Celestia).  However, for setiQuest work, I would prefer to use a Linux distro and coding tools that other folks here are using, for asking questions and comparing apples to apples.

What Linux distro and development tools are favored here at setiQuest?

Thanks,

-Don

maxs-pooper-scooper
Offline
Joined: 2010-07-28
Posts: 58
I agree Cygwin is probably not the best choice

and a virtual machine is better.

The default Linux they use is opensuse, but I got it to build on Fedora 14 VM ( see http://setiquest.org/forum/topic/open-sonata-source-code-released ).

Hope that helps

Don_G
Don_G's picture
Offline
Joined: 2011-12-15
Posts: 26
Thank you for this info maxs.

Thank you for this info maxs.