SonATA Porting Documentation

From setiquest wiki

Jump to: navigation, search

Contents

SonATA Porting Project:

SonATA Porting is a gsoc project that dealt with improving the build system, install instructions and porting the SonATA to various linux distros. Autotools was retained as the build system with modifications done to enable the new features offered by autotools.


A single top directory configure structure:

SonATA has five subdirectories which are essential to build and run it. Earlier to build SonATA a user had to generate configure for these directories and then configure then individually. And then make and install them separately. Now a user can generate the configure and run the configure at the top directory and run make and install from there.

It is not necessary that a user have to run the SonATA’s autotools build system from top. He can build individual packages also. This is one of the benefit of following nested approach. When only a sub-package has been changed. He can build only that.

Steps taken to implement::

  1. Basic configure.ac and Makefile.am was written. These generates Makefile.in and configure.
  2. Changes were made in the sig-pkg directory's configure.in ( renamed and updated to configure.ac) so that it can generate a common configure file which can generate a common Makefile. Now AC_CONFIG_FILES (earlier AC_OUTPUT) is used to generate these files instead of calling all the individual directories' configure.
  3. Disable the shared library in CppUnit to avoid conflict with tcl-readline.
  4. I have changed default prefix in tclreadline to $HOME/sonata_install

Control over nesting:

Sometime a developer might require that the individual directories of sig-pkg configure and make themselves. To include this a control over nesting was added. To use this feature a user can from the build directory give this:

./configure --enable-deepnesting

Steps taken to implement:

  1. AS_IF and AC_ARG_ENABLE are used to facilitate an option for this.
  2. See the section 'path to libraries and header files'.

VPATH BUILDS (Parallel Builds) :

Vpath builds refers to compiling the source code from different directory than build directory. Here source directory and build directory are different. This feature is enabled in autotools but some time its support can break.

Steps taken to re-implement this:

  1. Removing top_sourcedir from Makefile.am in sig-pkg. While compiling from source directory, top_builddir is equal to top_sourcedir are both equal to .. and builddir is equal to srcdir but when compiling from different directory it depends upon the location of source directory from build directory. Similarly I have removed the top_builddir as it is redundant because it is always equal to .. and will be given by configure.
  2. Swig in Makefile.am of sse-pkg/tsig and sse-pkg/ifc was also changed to give correct path to source files. In sse-pkg/tsig:
    from
    swig -c++ -tcl -o TestSig_wrap.cpp -ltclsh.i TestSig.i
    to:
    swig -c++ -tcl -o TestSig_wrap.cpp -ltclsh.i @srcdir@/TestSig.i
  3. Change the path to libraries and header files in Makefile.am. See section ' path to libraries and header files'.

A user now can keep his source directory clean by following the steps given below:

  1. cd SonATA
  2. mkdir objdir
  3. cd objdir
  4. ../autogen.sh
  5. ../configure
  6. make {optional options}
  7. sudo make install
  8. sudo chown -R `whoami` ~/sonata-install


Path to libraries and header files:

While path to libraries and header files work correctly currently but in newer implementation they build fails as make cannot find the libraries and header files. Following changes were made in path the path to libraries and header files in Makefile.am files in sig-pkg.


  1. Path to files was changed to use srcdir/../ instead of top_srcdir. This was done to implement control over nesting.top_srcdir is different when configuring via deep nesting. So path should be given through srcdir. Following are the changes that were applied in Makefile.am in different sub-directories of sig-pkg.
    1. -I$(top_srcdir)/include
      to
      -I$(srcdir)/../include
    2. -I$(top_srcdir)/lib
      to
      -I$(srcdir)/../lib
    3. PKT_INCLUDE = $(top_srcdir)/include
      to
      PKT_INCLUDE = $(srcdir)/../include
    4. PKT_DIR = $(top_srcdir)/../ATApackets
      to
      PKT_DIR = $(srcdir)/../../ATApackets
    5. SUP_INCLUDE = $(top_srcdir)/include
      to
      SUP_INCLUDE = $(srcdir)/../include
    6. SIGPROC_DIR = $(top_srcdir)/..
      to
      SIGPROC_DIR = $(srcdir)/../..
    7. SSE_INCDIR = $(top_srcdir)/../../sse-pkg/include
      to
      SE_INCDIR = $(srcdir)/../../../sse-pkg/include
    8. SSE_INTERFACE_INCDIR = $(top_srcdir)/../../sse-pkg/sseInterfaceLib
      to
      SSE_INTERFACE_INCDIR = $(srcdir)/../../../sse-pkg/sseInterfaceLib
    9. SSE_CHAN_INTERFACE_INCDIR = $(top_srcdir)/../../sse-pkg/sseChannelizerInterfaceLib
      to
      SSE_CHAN_INTERFACE_INCDIR = $(srcdir)/../../../sse-pkg/sseChannelizerInterfaceLib
    10. DFB_DIR = $(top_srcdir)/../dfbLib
      to
      DFB_DIR = $(srcdir)/../../dfbLib
    11. SSE_UTIL_INCDIR = $(top_srcdir)/../../sse-pkg/sseutil
      to
      SSE_UTIL_INCDIR = $(srcdir)/../../../sse-pkg/sseutil
    12. PKT_HDR_DIR = $(top_srcdir)/../ATApackets
      to
      PKT_HDR_DIR = $(srcdir)/../../ATApackets
    13. SONATA_DIR = $(top_srcdir)/../sonataLib
      to
      SONATA_DIR = $(srcdir)/../../sonataLib
    14. SSE_DIR = $(top_srcdir)/../../sse-pkg
      to
      SSE_DIR = $(srcdir)/../../../sse-pkg
    15. DFB_INCLUDE = $(top_srcdir)/include
      to
      DFB_INCLUDE = $(srcdir)/../include
    16. DFB_DIR = $(top_srcdir)/../dfbLib
      to
      DFB_DIR = $(srcdir)/../../dfbLib
    17. DADD_DIR = $(top_srcdir)/../daddLib
      to
      DADD_DIR = $(srcdir)/../../daddLib
    18. SPECTRA_DIR = $(top_srcdir)/../spectraLib
      to
      SPECTRA_DIR = $(srcdir)/../../spectraLib
    19. DX_INCDIR = $(top_srcdir)/include
      to
      DX_INCDIR = $(srcdir)/../include
    20. DX_SRCINCDIR = $(top_srcdir)/src
      to
      DX_SRCINCDIR = $(srcdir)/../src
    21. GAUSS_DIR = $(top_srcdir)/../gaussLib
      to
      GAUSS_DIR = $(srcdir)/../../gaussLib
    22. SONATA_DIR = $(top_srcdir)/../sonataLib
      to
      SONATA_DIR = $(srcdir)/../../sonataLib
    23. CHANNELIZER_INCLUDE = $(top_srcdir)/include
      to
      CHANNELIZER_INCLUDE = $(srcdir)/../include
    24. DX_INCLUDE = $(top_srcdir)/include
      to
      DX_INCLUDE = $(srcdir)/../include
  2. Similar changes were applied with respect to builddir and top_builddir.
  3. In many places in Makefile.am files under sig-pkg, srcdir is used instead of builddir. While this doesn't give error when source directory is same as build directory like in non VPATH builds, but in VPATH build make cannot find the source. builddir should be used when we need to link a library that is generated during build process and srcdir when headers files and source code files are needed.
    1. PKT_DIR = $(top_srcdir)/src
      to
      PKT_DIR = $(builddir)/../src
    2. SUP_DIR = $(top_srcdir)/src
      to
      PKT_DIR = $(builddir)/../src
    3. PKT_HDR_LIBDIR = $(PKT_HDR_DIR)/src
      to
      PKT_HDR_LIBDIR = $(builddir)/../../ATApackets/src
    4. DFB_LIBDIR = $(DFB_DIR)/src
      to
      DFB_LIBDIR = $(builddir)/../../dfbLib/src
    5. PKT_HDR_LIBDIR = $(PKT_HDR_DIR)/src
      to
      PKT_HDR_LIBDIR = $(builddir)/../../ATApackets/src
    6. SONATA_LIBDIR = $(SONATA_DIR)/src
      to
      SONATA_LIBDIR = $(builddir)/../../sonataLib/src
    7. SONATA_LIBDIR = $(SONATA_DIR)/src
      to
      SONATA_LIBDIR = $(builddir)/../../sonataLib/src
    8. DADD_LIBDIR = $(DADD_DIR)/src
      to
      DADD_LIBDIR = $(builddir)/../../daddLib/src
    9. DX_LIBDIR = $(top_srcdir)/lib
      to
      DX_LIBDIR = $(builddir)/../lib
    10. GAUSS_LIBDIR = $(GAUSS_DIR)/src
      to
      GAUSS_LIBDIR = $(builddir)/../../gaussLib/src
    11. SSE_DX_INTERFACE_LIBDIR = $(top_srcdir)/../../sse-pkg/sseDxInterfaceLib
      to
      SSE_DX_INTERFACE_LIBDIR = $(builddir)/../../../sse-pkg/sseDxInterfaceLib
    12. SSE_INTERFACE_LIBDIR = $(top_srcdir)/../../sse-pkg/sseInterfaceLib
      to
      SSE_INTERFACE_LIBDIR = $(builddir)/../../../sse-pkg/sseInterfaceLib
    13. PACKETSQIMPORT_LIB_DIR = $(top_srcdir)/lib
      to
      PACKETSQIMPORT_LIB_DIR = $(builddir)/../lib
    14. PACKETRELAY_LIB_DIR = $(top_srcdir)/lib
      to
      PACKETRELAY_LIB_DIR = $(builddir)/../lib
    15. PACKEKSEND_LIB_DIR = $(top_srcdir)/lib
      to
      PACKETSEND_LIB_DIR = $(builddir)/../lib
  4. Similarly in one case opposite scenario was found where builddir was used instead of srcdir.
    SSE_INTERFACE_DIR = $(top_builddir)/../../sse-pkg/sseInterfaceLib
    to
    SSE_INTERFACE_DIR = $(srcdir)/../../../sse-pkg/sseInterfaceLib

Check for header files and libraries

Steps taken to implement the check:

  1. I have added a macro (AX_EXT_HAVE_LIB) implement the checks.I have modified it so that it reports error when it can't find the libraries.
  2. Following libraries are searched and if not found error is reported: mysqlclient, fftw, mpfr gmp and ACE libraries in non standard locations also. This solve the linker error that arises when a library is in non standard location ( as is the case with fedora in case of mysqlclient).
  3. Also following headers are checked:arpa/inet.h fcntl.h float.h inttypes.h limits.h netdb.h netinet/in.h stdlib.h string.h strings.h sys/file.h sys/ioctl.h sys/socket.h sys/statfs.h sys/statvfs.h sys/time.h unistd.h utime.h .

Updating the build system

SonATA build system was updated in following ways:

  1. Now it is autoconf version 2.63 compliant.
    • This version is better documented.
    • Many of the obsolete macros were changed in configure.ac files.
      1. AC_OUTPUT is used differently now along with AC_CONFIG_SUBDIRS.
      2. AC_INIT too has been changed.
      3. LIBTOOL is now used using LT_INIT instead of deprecated AC_LIBTOOL_DLOPEN AC_PROG_LIBTOOL
  2. Silent rules was added. This is conditional. If it is found then it can be used otherwise it can't.To use this compile using: make V=0 .
    Silentrulescompilation alt text
  3. I have updated the search path in tclreadline configure.ac so that it can found tcl related libraries and header files automatically.

All these and similar changes can be easily noted by comparing the configure.in and the configure.ac (the recommended) files.

Porting to the Distros:

The SonATA was ported to OpenSuse 11.4, Fedora 15, Fedora 14, Ubuntu 11.04, Debian 6 squeeze and Centos 5. I have been able run SonATA successfully. While it is yet to tested by anyone else beside me on Ubuntu and Centos. I doubt there will be any trouble if instructions given by me are followed. In Fedora there was activity failure when test was done. Further testing by others will hopefully show positive results.


OpenSuse 11.4:

Steps taken to port:

  1. Building a rpm metapackage by creating a spec file.
  2. Writing a script to automate most of the pre-build steps...


Issues faced:

  1. OpenSuse 11.4 require update otherwise Eperm error arises when scheduling.


Steps require to build and test now:

  1. Edit the /etc/sudoers file and add the following line as the last line:
    {username} ALL=(ALL) NOPASSWD: ALL
  2. Run the script suse_preins.sh
  3. Then go into Sonata directory:
    cd ~/SonATA
  4. Then issue following script and commands:
    1. ./autogen.sh
    2. ./configure
    3. make
    4. sudo make install
    5. sudo chown -R `whoami` ~/sonata-install
  5. To test type the following in a terminal:
    test_sonata
  6. Then after channelizer and dx have connected. Type the following in seeker window:
    source vger-demo-xpol.tcl
  7. Type the following in another terminal:
    waterfallDisplay

Debian 6 (Squeeze):

Steps taken to port:

  1. Created a meta-package which installs all the dependencies needed in debian.
  2. Writing a script to automate most of the pre-build steps.


Issues faced:

  1. Debian uses threaded tcl due to which malloc and Tcl_Alloc cannot be interchanged. Also free and Tcl_Free cannot be interchanged.
    This issue arises in sse-pkg/sse/readmask.cpp and tclreadline/tclreadline.c
  2. Debian doesn't support SIGILL trapping using expect script as it is un stable. So it has to be disabled from sse-pkg/util/controlcomponent.expect file.
  3. Debian uses tail which is POSIX 2001 compliant. So tail gives script gives error when use in it. In OpenSuse this doesn't arises as its utilities are not POSIX 2001 compliant.
  4. Sudo cannot find the path of packetsend or any path set in the bash.rc.


Steps require to build and test now:

  1. Change the following line in SonATA/scripts/runpacketsend-dx-vger-xpol.tcsh:
    sudo packetsend -c -f ${HOME}/sonata_install/data/vger-xpol-2010-07-14-406.pktdata -J 229.1.1.1 -j 51100 -n 1 -i 1050 -b 1
    to
    sudo env PATH=$PATH packetsend -c -f ${HOME}/sonata_install/data/vger-xpol-2010-07-14-406.pktdata -J 229.1.1.1 -j 51100 -n 1 -i 1050 -b 1
  2. Edit the /etc/sudoers file and add the following line as the last line:
    {username} ALL=(ALL) NOPASSWD: ALL
  3. Run the script sonata_i.sh
  4. Then from go into Sonata directory:
    cd ~/SonATA
  5. Then issue following script and commands:
    1. ./autogen.sh
    2. ./configure
    3. make
    4. sudo make install
    5. sudo chown -R `whoami` ~/sonata-install
  6. To test type the following in a terminal:
    test_sonata
  7. Then after channelizer and dx have connected. Type the following in seeker window:
    source vger-demo-xpol.tcl
  8. Type the following in another terminal:
    waterfallDisplay

Ubuntu (Natty):

Steps taken to port
  1. Debian's metapackage is compatible with Ubuntu.
  2. Writing a script to automate most of the pre-build steps.
  3. Partner repository has to be enabled to install sun-java.
Issues faced
Steps require to build and test
  1. Change the following line in SonATA/scripts/runpacketsend-dx-vger-xpol.tcsh:
    sudo packetsend -c -f ${HOME}/sonata_install/data/vger-xpol-2010-07-14-406.pktdata -J 229.1.1.1 -j 51100 -n 1 -i 1050 -b 1
    to
    sudo env PATH=$PATH packetsend -c -f ${HOME}/sonata_install/data/vger-xpol-2010-07-14-406.pktdata -J 229.1.1.1 -j 51100 -n 1 -i 1050 -b 1
  2. Edit the /etc/sudoers file and add the following line as the last line:
    {username} ALL=(ALL) NOPASSWD: ALL
  3. Run the script ubu_preins.sh
  4. Then from go into Sonata directory:
    cd ~/SonATA
  5. Then issue following script and commands:
    1. ./autogen.sh
    2. ./configure
    3. make
    4. sudo make install
    5. sudo chown -R `whoami` ~/sonata-install
  6. To test type the following in a terminal:
    test_sonata
  7. Then after channelizer and dx have connected. Type the following in the seeker window:
    source vger-demo-xpol.tcl
  8. Type the following in another terminal:
    waterfallDisplay

Centos 5:

Steps taken to port:

  1. Centos metapackage was built by writing a spec file.
  2. Writing a script to automate most of the pre-build steps.


Issues faced:

  1. Apart from the issues that arised in Debian Centos has following issues:
  2. No repository of Centos offered the sun-java.
  3. In sse-pkg/ifc/ifc.i and sse-pkg/tsig/TestSig.i, the blank line above the #endif causes error.(Due to the version of swig used).
  4. Fftw and other libraries offered by repositories are incompatible with SonATA.
  5. Default C flags are incompatible with centos gcc versions.
  6. Compiled ACE_wrappers given is incompatible with Centos 5.
  7. Mysqlclient libraries path is not standard due to which a linker error arises when compiling (this was solved when I wrote the check for libraries).


Steps require to build and test:

  1. Change the line following line in sse-pkg/util/controlcomponents.expect from:
    trap quit {INT TERM QUIT ABRT ILL HUP }
    to
    trap quit {INT TERM QUIT ABRT HUP }
  2. Change the 52 and 53 line in SonATA/tclreadline/tclreadline.c from:
    #define MALLOC(size) Tcl_Alloc((int) size)
    #define FREE(ptr) if (ptr) { Tcl_Free((char*) ptr); ptr = 0; }
    to
    #define MALLOC(size) malloc((int) size)
    #define FREE(ptr) if (ptr) { free((char*) ptr); ptr = 0; }
  3. Change the lines in sse-pkg/sse/readMask.cpp :
    free((char *) maskv);
    to
    Tcl_Free((char *) maskv);
  4. Remove the blank lines above the #endif from sse-pkg/ifc/ifc.i and sse-pkg/tsig/TestSig.i
  5. Edit the /etc/sudoers file and add the following line as the last line:
    {username} ALL=(ALL) NOPASSWD: ALL
  6. Run the script cen5_preins.sh
  7. Then from go into Sonata directory:
    cd ~/SonATA
  8. Then issue following script and commands:
    1. ./autogen.sh
    2. ./configure
    3. make CXXFLAGS='-Wall -Woverloaded-virtual -pthread -fno-strict-aliasing -msse3'
    4. sudo make install
    5. sudo chown -R `whoami` ~/sonata-install
  9. To test type the following in a terminal:
    test_sonata
  10. Then after channelizer and dx have connected. Type the following in seeker window:
    source vger-demo-xpol.tcl
  11. Type the following in another terminal:
    waterfallDisplay

Fedora 15:

Steps taken to port:

  1. Fedora metapackage was built by writing a spec file.
  2. Writing a script to automate most of the pre-build steps.


Issues faced:

  1. No repository of fedora offers the sun-java.
  2. Mysqlclient libraries path is not standard due to which a linker error arises when compiling (this was solved when I wrote the check for libraries).
  3. Firewall block the connection to port 51110 udp due to which activity fails.
  4. Compiling fails for SonATA unless -fpermissive flags is given in gcc 4.6.
  5. During runtime sudo cannot find the path if default configuration of /etc/sudoers is used due to following lines:
    Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin


Steps require to build and test:

  1. Open the port udp 51110 or disable the firewall using system-config-firewall .
  2. Edit the /etc/sudoers file and add the following line as the last line:
    {username} ALL=(ALL) NOPASSWD: ALL
  3. Comment the 'Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin' line in /etc/sudoers.
  4. Run the script ubu_preins.sh
  5. Then from go into Sonata directory:
    cd ~/SonATA
  6. Then issue following script and commands:
    1. ./autogen.sh
    2. ./configure
    3. make
    4. sudo make install
    5. sudo chown -R `whoami` ~/sonata-install
  7. To test type the following in a terminal:
    test_sonata
  8. Then after channelizer and dx have connected. Type the following in seeker window:
    source vger-demo-xpol.tcl
  9. Type the following in another terminal:
    waterfallDisplay

SonATA running on Fedora 15

Porting to the 32 bit Distro

SonATA was successfully ported to a 32 bit Ubuntu. The only issue that arised was related to ACE_wrapper compilation. While ACE_wrapper is also compiled during Centos 5 but that method fails due to a bug in ACE_wrapper build system while compiling in 32 bit. Steps required to compile ACE_wrapper on 32 bit:

  1. In ACE_wrappers/include/makeinclude/platform_linux.GNU , change the line
    LIBS += $(shell test "`ls -L /usr/lib*/librt.so* /lib*/librt.so*`" && echo -lrt)
    to
    LIBS += $(shell test "`ls -L /usr/lib*/librt.so* /lib/*/librt.so* /lib*/librt.so*`" && echo -lrt)
  2. In ACE_wrappers/include/makeinclude/platform_linux_common.GNU, change the line $(shell test "`ls -L /usr/lib*/librt.so* /lib*/librt.so*`" && echo -DACE_HAS_AIO_CALLS)
    to
    $(shell test "`ls -L /usr/lib*/librt.so* /lib/*/librt.so* /lib*/librt.so*`" && echo -DACE_HAS_AIO_CALLS)
  3. In ACE_wrappers/MPC/templates/makedll.mpt, change the line
    ldlibs = -ldl $(subst lib,-l,$(sort $(basename $(notdir $(wildcard /usr/lib/librt.so /lib/librt.so))))) -lpthread
    to
    ldlibs = -ldl $(subst lib,-l,$(sort $(basename $(notdir $(wildcard /usr/lib/librt.so /lib/*/librt.so /lib/librt.so))))) -lpthread
  4. From ACE_wrapper directory:
    1. dos2unix *
    2. cd aux_config
    3. cd ..
    4. dos2unix bin/bootstrap
    5. changing the format of configure to unix using :set fileformat=unix in vi.
    6. rm lib/*
    7. rm ace/*.so*
    8. ./bin/bootstrap
    9. mkdir objdir && cd objdir
    10. ../configure --disable-gperf --enable-ssl=no --prefix=$HOME/SonATA/packages/ACE_wrappers --enable-tests=no
    11. make
    12. make install
    13. cd ..
    14. cp lib/* ace/

Also xtail was compiled.

They are available in binary form as packages_32bit_1.0.tar.gz .

SonATA running on 32 bit Ubuntu

Build and test instructions for 32 bit Ubuntu is same as on 64 bit except that you have to use the script ubu_preins.sh from folder 32.

Merging The Project

  1. All the changes can safely applied. I have created a separate gsoc branch. And then I applied it.
  2. But in the pre-install scripts following changes will have to be applied to the scripts:
    • Locations of meta-packages given to wget in these scripts. Currently they are at gsoc folder in the gsoc branch
    • Scripts clone the sonata from my gsoc branch when SonATA is not found. That too will have to be amended.
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox