Talk:SonATA Porting

From setiquest wiki

Jump to: navigation, search

Back to project page

Khrm - We can discuss project items here. I am watching this page, so if you make an edit I will get an email. make sure to select the "Watch this page" checkbox below. -jrseti

Contents

32-bit Linux support?

What do you think about supporting 32-bit Linux as well as the current 64-bit Linux? Is 64-bit addressing used anywhere in SonATA or is there some other 64-bit technical problem? The reason I mention 32-bit is because most people install 32-bit Linux by default even though they have 64-bit hardware. - SigBlips 16:38, 10 May 2011 (PDT)

Maybe we can try compiling and running on a 32 bit computer and see what the issues are. 32 bit will be slower and there are issues with the software that if the data is not processed fast enough the software does not work. I would like to address these speed issues first before the 32 bit. Jrseti 16:46, 10 May 2011 (PDT)
I think two main issues are that current ace wrapper's binary given is for 64 bit and 4 gb ram can't be used by the user unless they have PAE enabled.KHRM 02:22, 11 May 2011 (PDT)
Do any SonATA processes use 4 GB of RAM? I don't know this answer but my guess would be no. The ATA data is sampled by 16-bit ADCs at 105 Msamples/second which is 210 MB/sec. Calculation 4000 MB / 210 MB/s = 19 seconds. Due to the real-time nature of SonATA I doubt that any process is holding 19 seconds of full bandwidth data at any particular time. - SigBlips 10:37, 11 May 2011 (PDT)
But 4gb is the minimum requirement to run SonATA. On 3gb we might be able to run it but then it will be quite slow.KHRM 09:05, 12 May 2011 (PDT)

Which Linux Distributions to target?

Which distributions should we target? I think we can add Ubuntu,Debian,Fedora,Centos to the list. Any more OS apart from them?KHRM 09:05, 12 May 2011 (PDT)

I have a question about the http://setiquest.org/wiki/index.php/File:LinuxDistros.png chart. Is the vertical axis percentage? The chart says "Usage of Linux for websites" so I'm guessing the context is massive server farms like AWS and other hosting companies. If so then this is a very different Linux distribution mix than what people develop on and and use on the desktop. - SigBlips 10:41, 12 May 2011 (PDT)
Actually that scenario is not much different. Lets take Linux Mint which has quite a substantial base in desktop but then it is just a Ubuntu. Same with other derivatives.Even repositories are same or compatible with each other.So instructions will be same.KHRM 07:24, 13 May 2011 (PDT)
The default AWS Linux AMI is based on a very stripped down CentOS 5. Basic things like gcc and autotools will need to be installed but that is trivial with yum. CentOS is basically a free fork of Red Hat Enterprise Linux (RHEL). So supporting the default AWS AMI shouldn't be any extra work if RHEL is being targeted. AWS also makes for a great remote test environment. - SigBlips 15:36, 28 May 2011 (PDT)
You are quite right. And some other error that I encounter were in ifc.i and tsig .i as they have a gap of blank line between #endif. Also ACE_WRAPPER's libace.so have to be compiled as it is not compatible with one given. Also fftw3 distributed is not compatible with. There's no -lfftwf ( or something similar). So you will have to compile your own. And when you start to compile it it complains that autotools is old.So my current script look like https://gist.github.com/1078489 for centos. It is quite challenging to install in that. But hopefully my script will automate most of the process.KHRM 10:38, 12 July 2011 (PDT)
I'm really confused after yesterdays IRC community meeting. KHRM mentioned Debian was working but there was talk about OpenSUSE 11.4 problems. What Linux distributions are the SonATA port currently working with? Last month I thought Ubuntu and/or Fedora were working. What Linux distributions are being worked on currently? Which ones are on the TODO list? We probably should revamp the SonATA Porting page to better reflect the past, work in progress, and future Linux distribution / version porting status. - SigBlips 10:40, 6 July 2011 (PDT)
It works in OpenSuse11.4 after updating the fresh install. Since I have already updated it I didn't encounter any error but when a fresh install was done, it give error related to scheduling of threads permission. Also it works in fedora(14 I will try the newer 15 that has come recently) but you have to install sun-java separately.-khrm 120.56.178.105 11:39, 6 July 2011 (PDT)

Today I will finish writing ubuntu's automate script ( just change in 3 lines or one step from Debian) and Fedora's script. So we will complete OpenSuse 11.4, Debian, Ubuntu, Centos 5 and Fedora 15. This week I will target Gentoo.KHRM 19:20, 17 July 2011 (PDT)

Earlier in the script I was asking the user to add the new repo but now I have removed that. Now I create the backup of current repo and use the repo needed to download the dependencies. And then restored the old repo.
Did you use CentOS 5.6? Which versions of Debian and Ubuntu did you use? The most recent ones? It might interesting to try a several year old version of Ubuntu. - SigBlips 20:07, 17 July 2011 (PDT)
I install using net install method. I use this mirror:ftp://mirror.csclub.uwaterloo.ca/centos/5/os/x86_64/CentOS/ It's Centos 5. It has packages from 2 months-3 years old.And I used the rpmforge 5 repo ( it is same for all minor versions whether 5.6 or 5.1 ). Also I found out except some minor packages, in a minor versions all the packages are of same versions ( like kernel 2.6.18 and an older glibc and other libraries). For I used Debian Squeeze and Ubuntu 10.10 ( I have to check on 11.04).KHRM 10:38, 18 July 2011 (PDT)

Added discussion Section

Khrm: I added a discussion area to the project WiKi page. See [[1]]. I added specifics about which Linux distributions to target, in what order. Also, I started the discussion about how to improve the build system. Have a look, comment.

I think we should first concentrate on improving the build system in the existing Suse version we already have. Then, once that is perfected we can start the move to other OS's. Jrseti 10:18, 12 May 2011 (PDT)

I'm confused. Why do you want a discussion section in the content page when we have a nice discussion in the Talk: page here? What sort of discussion goes where? Did you know that you can post images in a Talk: page? - SigBlips 10:33, 12 May 2011 (PDT)
My idea was that the results of the discussions would go in the content page, but the actual discussion would go here. The content page would quickly get too full. Does this not make sense?Jrseti 11:31, 12 May 2011 (PDT)
It appears that the Linux distribution discussion is occurring in two different places at the same time. Where do I comment? Here or there? General wiki protocol is that editing any contents of the main page is fair game but user comments in the Talk: discussion pages are considered sacred and are not to be modified by other users except for minor formatting fixes. So can I edit your comment at SonATA_Porting#Which_Linux_distributions_should_we_target? It wouldn't seem polite if I did. - SigBlips 12:25, 12 May 2011 (PDT)
I think it will be better if we group these distributions. 1.Debian and Ubuntu 2.Fedora,RHEL and CentOS 3.OpenSuse 4.Gentoo. As distributions in each groups take from each other/are derivatives of another.KHRM 07:17, 13 May 2011 (PDT)
Grouping Linux distributions by genealogy is a good idea and it makes a lot of sense. What do you think of the idea of also porting SonATA to FreeBSD, Mac OS X, and Solaris? I've found pthreads to be very compatible across all of those platforms. The hard part will be porting to the first non-Linux OS, after that the others should be fairly easy from a SonATA perspective. - SigBlips 13:30, 13 May 2011 (PDT)
FreeBSD is freely available. But I think they are all have different libc implementations which should also be taken into account but yes when it has been build to a different OS then it can be easily build for another.KHRM 09:44, 14 May 2011 (PDT)
OpenSolaris runs on x86_64 and is also freely available. The SETI Institute has a number of old Sun Solaris SPARC machines laying around that could be recruited for this port. Many SETI Institute employees also use Apple Macintoshes so a Mac OS X port might be valuable to them too. The benefits of porting SonATA to a couple non-Linux operating systems and a non-Intel x86 architecture would be improved code robustness. It is also a great way to find bugs. - SigBlips 11:10, 20 May 2011 (PDT)
There should be cpu with sse which I think is available only in amd and intel.KHRM 07:55, 22 May 2011 (PDT)
Yes you are correct, only recent Intel and AMD chips support SSE3. It should be possible for autotools or cmake to detect the CPU type and then set conditional #ifdef compilation flags. Every section of SonATA SIMD code should have a plain C code pair. I think a good project goal is for SonATA to be able to compile on any architecture. I mean with/without SIMD optimizations, big/little endian, and 32/64-bit CPU modes. - SigBlips 19:03, 22 May 2011 (PDT)
This is quite possible.KHRM 09:58, 23 May 2011 (PDT)

Added issue tracker

I added an issue tracker for this project at [[2]].Jrseti 11:34, 12 May 2011 (PDT)

Great. I added that link and two GitHub links to the new SonATA_Porting#External_links section. The general idea of the External links section is to be a quick jumping point to other non-wiki pages that are directly related to the management of this project. - SigBlips 13:38, 12 May 2011 (PDT)
I added Sigblips as a developer to http://issues.setiquest.org/projects/sonata-porting. Feel free to ad, assign, comment, whatever you wish. I would have done this earlier, but i had trouble figuring out how to add people to projects as developers. Now i know! Jrseti 09:19, 26 May 2011 (PDT)
KRHM: Start using http://issues.setiquest.org/projects/sonata-porting to enter each separate issue you are working on. Also, enter in the estimated time you think it will take, and also enter the time it actually took. Note that you will not be judged negatively if you, for instance, say something will take you 5 hours but it actually takes you a lot more, like 30 hours. We would like to start learning how to estimate time better and this may help us learn. Jrseti 09:19, 26 May 2011 (PDT)

How are we going to improve the build system?

Continuing the discussion from the main page.

I think we can create a meta-packages for different distributions and distribute them. Or if we are talking about the different packages that need to be built to run SonATA then I think we can combine the build process of all these while keeping them independent.

Cmake can check the packages and report it if missing but I have not seen any example of cmake installing dependencies.KHRM 11:53, 13 May 2011 (PDT)

You might find this old GiD party install script on GitHub [3] useful. It was used to automatically install the packages that SonATA needed. It had problems and you'll need to ask jrseti exactly why it was removed. I mention it because the idea is potentially useful. - SigBlips 14:05, 13 May 2011 (PDT)
The reason I stopped using this script is that I found it was much more convenient to use zypper to install the necessary packages, rather than using this script.Jrseti 07:03, 17 May 2011 (PDT)
The software install by that script might conflict with the software install by user using official repositories through zypper which is the norm.KHRM 10:02, 18 May 2011 (PDT)
My first exposure to cmake was recently in the Algorithms project. It does seem much easier to configure than autoconf but I do see some potential problems. It is not standard on all Linux distributions so it will be one more thing many users will need to install. Cmake is an out-of-source build system which is very different than what SonATA currently uses. It is cleaner in that the source code files are separate from the object and binary files but this new directory hierarchy also makes it different. Has the SonATA team looked at some example cmake projects to see if this is a change they would like? - SigBlips 07:35, 14 May 2011 (PDT)
Actually this should work on autotools also by default if automake files have been written correctly.This is called vpath or parallel build in autotools. You can have same build and source directory or different. This has been rectified by me.KHRM 16:09, 10 June 2011 (PDT)
One of the major project using is KDE. Users will not have to install anything more than Cmake. In the current build system user have to install autotools components too.KHRM 09:37, 15 May 2011 (PDT)
So autotools is an optional install on OpenSuSE Linux 11.3? It has been a default install on all of my many Linux systems. Autotools is an optional install on AWS Linux but that distro is so stripped down it doesn't even have gcc. Cmake is new and if it gains enough popularity it will attain this same level of status in time. This isn't a real problem though, just an interesting difference. - SigBlips 08:32, 17 May 2011 (PDT)
Many distributions to save space keep out the all build tools (even gcc) from the live cd .But dvd versions have these.KHRM 12:05, 17 May 2011 (PDT)
I have encountered systems that need autotools installed. I think it all depends on how the person installed the OS, what software packages they selected. Also, I've had trouble in the past with systems where the autotools install is out of date and the build fails miserably. That gets really confusing. Jrseti 08:42, 17 May 2011 (PDT)
Do you think we should list a minimum required version of autotools (or future cmake) at SonATA#Dependencies? - SigBlips 08:53, 17 May 2011 (PDT)
It seems livecd version of OpenSuse didn't have even make installed.KHRM 09:58, 23 May 2011 (PDT)
So, what does CMake buy us? Is there any reason to do the work to move over to CMake? Or, would it be better to keep autoconf? Can we make the configure script we now use check for all the necessary packages?Jrseti 07:04, 17 May 2011 (PDT)

I would suggest we create software repositories for each OS (this is what Sigblips called "meta-packages" above). That would make it easier and we could ensure all the necessary packages are installed. Then change the configure script to complain if any necessary pages are missing. Currently we have 2 main directories that need to be built. sse-pkg, sig-pkg. The user has to change to these directories and compile separately. What we need is a set of configure/make files at the directory above sse-pkg and sig-pkg that will build both of these packages. This would save the user an extra building step.Jrseti 07:04, 17 May 2011 (PDT)

Actually by meta-packages I meant packages that don't install any software but instead depends upon others to ensure that they are installed. Lets we say need B and C. So we create a meta-package see A.deb/rpm which depends upon B and C. So it will install B and C but it will not install any files of its own.KHRM 08:59, 17 May 2011 (PDT)
Do you know how to create the meta-package? Have you done this before?Jrseti 16:38, 19 May 2011 (PDT)
No I have not done it before. But I can.KHRM 07:55, 22 May 2011 (PDT)

After building the meta package I think we can expand on it and can install scripts in the /usr/local/bin

  1. sonata_ic.sh which automatically downloads the SonATA using git and download extra libraries and data (like voyager) and prepare several files. Then we will be left with account setup, ssh step, build and test run.
  2. sonata_im.sh which is same as above but it assumes that users have already downloaded the sonata.KHRM 09:58, 23 May 2011 (PDT)
I have build a single script which does the work of above.It is in my gsoc branch: https://github.com/khrm/SonATA/blob/gsoc/gsoc/sonata_i.shKHRM 14:43, 25 May 2011 (PDT)

reducing RAM requirement

The first step in reducing SonATA's 4 GB RAM requirement is knowing how much memory each process is using. Can you run the Voyager test script and halfway through processing do a "ps auxw" and post the text here? It will be interesting to see CPU usage too. I don't have a machine with the specs to run SonATA so I can't do this myself. - SigBlips 18:39, 30 May 2011 (PDT)

I agree with Sigblips' in that we should figure out how much memory and CPU each process is using. Jrseti 06:20, 31 May 2011 (PDT)
Below is a <nowiki> formatting example. Could someone please paste the real process text output here? The output from the "top -H" command (H is for threads) might be even more useful. Thanks. - SigBlips 07:17, 31 May 2011 (PDT)
I have posted it here: https://gist.github.com/1003273 .We can see that top three are packetsend, channelizer and then dx.I will post the output with -H.KHRM 13:43, 1 June 2011 (PDT)
I have extracted the SonATA related processes that are using a significant amount of system resources:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
khrm      4778  0.4  1.4 1827652 30624 pts/3   Sl   05:39   0:20 java -jar WaterfallDisplay.jar
khrm      8147  0.0  0.1 357024  3128 pts/5    Sl+  06:42   0:00 seeker
khrm      8164  0.0  0.0 116656  1420 pts/11   S+   06:42   0:00 expect controlcomponents.expect
khrm      8167  0.0  0.0  34124  1056 pts/11   S+   06:42   0:00 dxArchiver -dx-port 8857
khrm      8169  0.0  0.0  34124   192 pts/11   S+   06:42   0:00 dxArchiver -dx-port 8858
root      8235  0.2  9.2 2197860 189672 pts/13 Sl+  06:42   0:02 channelizer
root      8652 25.8 20.1 1550220 411852 pts/12 Sl+  06:52   0:58 dx
root      8710 11.2 49.9 2704224 1022892 pts/5 Sl+  06:52   0:21 packetsend -c -f vger -i 1050 -b 1

Note that the test machine had 2 GB of RAM which resulted in some swapping. This means that the resident set size (RSS) memory values fluctuated a bit and are not 100% accurate. Also note that the java start time and pid are low so the Waterfall Display was likely launched prior to this test run. It is interesting that the seeker is the most complicated and important part of SonATA yet it consumes very little CPU and memory resources. - SigBlips 15:19, 1 June 2011 (PDT)
Sigblips: You are correct, the seeker is not really doing much. It mainly directs the various components to do their thing and keeps all working together. The programs that actually handle the data are the problem. 10.1.1.130 21:19, 22 June 2011 (PDT)
So, what does this tell us we need to do? Can we make these programs allocate only 2GB and no more without swapping? 10.1.1.130 21:19, 22 June 2011 (PDT)
I think the next step should be determining why packetsend is using 50% (1 GB) of RAM. I am guessing that it is preloading the entire 624 MB vger-2010-07-14-406.pktdata file into RAM. It should be possible to reduce packetsend's memory footprint to just a couple megabytes. Getting a good snapshot of /swap usage would be good to know too. - SigBlips 09:56, 23 June 2011 (PDT)
You are right. If we somehow get a packetfile of 100 mb only instead of 600 mb of voyager data we will have RAM reduced by 500mbs. To test this I use the demo-testfile.pktdata available in SonATA/script package. RAM usage was just 444 mb which includes 37 mb of packet data.
sudo $HOME/packetsend -c -f  $HOME/SonATA/scripts/demo-testfile.pktdata -J 229.1.1.1 -j 51100 -n 1 -i 1050 -b 1

KHRM 06:16, 21 July 2011 (PDT)
That was a great test. It verifies the location of the excessive RAM usage with certainty. Now the question is if it's easier to fix packetsend using the existing library infrastructure or to rewrite it? - SigBlips 10:53, 25 July 2011 (PDT)

UDP buffer overflow

One problem that I've encountered is that when SonATA sends data packets over UDP it does it as fast as possible. There are no checks to see if socket buffer is full and wait till it has enough room. Thus, if your system does not have enough CPU or the network drivers can not keep up, things stop working. Look at the packetread program where it sends the data packets over the network. Instead of just sending the data blindly, we really should be checking to see if the socket buffer has room. Also: If we do make these changes to check for room in the socket buffer before sending, we need to surround these changes with #ifdef statements so the default can be to compile without checking. Jrseti 06:18, 31 May 2011 (PDT)

I think trouble is with packetsend which does the above. Packetread I think is reading the data as fast as possible.I might be wrongKHRM 09:08, 1 June 2011 (PDT).
Yes, I think he meant packetsend which does send UDP packets as fast as possible unless the -i delay useconds flag is used. Looking at the fullness of packetsend's output buffer won't be that helpful. What you want to do is monitor the fullness of the recv() buffers in SonATA. Unfortunately this will be difficult for a variety of reasons. Throttling with some sort of handshake will be a more robust solution but it sort of defeats the point of using multicast UDP in the first place. - SigBlips 11:45, 1 June 2011 (PDT)
Yes, I get the names confused sometimes. packetsend does have the delay option, but how can you tell if you have the delay set large enough? I think we should be looking for the fullness of packetsend's output buffer. That may make things a bit better. Right now I think the program quits or hangs if the buffer gets full, so that is not a good way to handle a full buffer. But you are correct in a way, the other problem is the recv() buffers in the dx and channelizers, as well as the send() buffer in the channelizer. This will be difficult to do because there is no back method of communication to tell packetsend to slow down or pause for catchup. So maybe a good simple approach is to make sure the packetsend, dx and channelizer programs gracefully handle the buffer full conditions, send() and recv() without crashing. Maybe they could print out some information informing the user that this is happening and suggest that packetsend be slowed down with the packetsend delay option increased. 10.1.1.130 21:31, 22 June 2011 (PDT)

configure.in CXXFLAGS not global

The feature http://issues.setiquest.org/issues/276 should be moved from the main SonATA project to the SonATA Porting project. I don't have the manager permissions to do this. - SigBlips 12:18, 9 June 2011 (PDT)

A user can give the flags during make.KHRM 15:11, 13 June 2011 (PDT)
I tried setting the flag but it didn't seem to work. Either I did it wrong or the autotools script is broken. The bigger picture view is that autotools should determine the version of gcc and not try to use incompatible compiler flags. If we want a portable make system then that will be required. - SigBlips 20:14, 13 June 2011 (PDT)
I gave make CXXFLAGS='-Wall -Woverloaded-virtual -pthread -fno-strict-aliasing'. Then flags -msse3 and -march=native were not used. Here are two cases https://gist.github.com/1036570 and https://gist.github.com/1036567 .KHRM 14:03, 20 June 2011 (PDT)
That didn't seem to work when I tried it. Maybe I did it wrong. I'll try it again and let you know. Thanks. - SigBlips 11:54, 21 June 2011 (PDT)

jrseti Testing results

10.1.1.130 21:14, 22 June 2011 (PDT)

I did some testing using your latest from GitHub. Here are some things I found:

  1. At the end of creating the keys there is an error message that says "cp: cannot stat './id_rsa.pub' no such file or directory. " The authorized_keys file does not get created, but after there is a .ssh/id_rsa.pub file, so you may be doing something in the wrong order?
  2. The sshd daemon is not running on new systems. Maybe you should add instructions on this at the end of the install.
  3. Doing a "sudo make install" does not put the files into sonata_install. It should, right? It puts them in .usr/local. We need to have the files install into ~/sonata-install.
  4. Remember Josh complained about the errorhe got while unzipping the voyager data file? I get that error also. Either the file is corrupt on the server or there is a problem with gunzip on OpenSUSE 11.4
  5. Remember Josh complained that autogen and libtoolize were not on his system? I had to install those on his system with zypper. On my OpenSUSE install, these existed. So I think this was the fault of how the OS was installed. We will want to add tis information in the install instructions
  1. That error occured because some distro (11.3) have default key as ./id_rsa.pub and other as ./ssh/id_rsa.pub . So to solve this I am using ssh-keygen -f sonata -t rsa -q and ssh-copy-id -i ~/.ssh/sonata.pub $USERNAME@`hostname` in script now.
  2. Even after using the metapackage? I think after that it should start by default. If it didn't then I will include that.
  3. I have failed to detect it as I had install in /sonata_install using prefix $HOME/sonata_install. Now I have changed the default prefix to that.
  4. Did you check the md5sum? I have install OpenSuse 11.4 and it is working for me. This error occurs when an non gzip package is given.
  5. I think metapackage should install that also.KHRM 11:37, 23 June 2011 (PDT)

Documentation:

I have written documentation on the work that I have done here: http://setiquest.org/wiki/index.php/SonATA_Porting_Documentation ( I am still writing).KHRM 12:50, 21 August 2011 (PDT)

I have yet to finish the http://setiquest.org/wiki/index.php/SonATA_Porting_Documentation#Check_for_header_and_libraries, http://setiquest.org/wiki/index.php/SonATA_Porting_Documentation#Updating_the_build_system and http://setiquest.org/wiki/index.php/SonATA_Porting_Documentation#Path_to_libraries_and_header_files: . I am using http://extensions.services.openoffice.org/en/project/wikipublisher KHRM 12:59, 21 August 2011 (PDT)
I have completed the documentation.KHRM 08:18, 22 August 2011 (PDT)


I have merge the gsoc branch to master. https://gist.github.com/1163121 Here's the diff between master and gsoc.KHRM 11:36, 22 August 2011 (PDT)

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox