For now we're going w/ TravisCI. Ideally, I'd like a Jenkins server, but I don't have a good place to host that ATM.

We have yet to get root installed, running, and tested on the CI server.

I'm having trouble cracking how to write to a memory buffer instead of to a file. This works with stringstreams if I just use fundamental types and strings. I can't get it to work with vectors or other binary data types. The current implementation writes a file to the disk, reads the file back and then removes the file.

Unit tests Affected:

• unittest-HelperFunctions
• unittest-{Data, Pld}Writer
• unittest-{Data, Pld}Reader

Introduction

This issue will serve as a simple logbook for creating the MVP data generator.

The Goals

The goal is to write a program that

1. generates some static XiaData classes.
2. writes those data to a PLD and an LDF

Setup

Here are the values that are hard coded into the generator.

    const unsigned int channelNumber = 0;
    vector<unsigned int> encodedData;
    const unsigned int energy = 55;
    const string firmware = "30474";
    const unsigned int frequency = 250;
    const unsigned int ldfFormatCode = 0;
    const unsigned int moduleNumber = 0;
    const unsigned int numberOfDataWords = 4;
    const unsigned int numberOfDataBuffers = 2;
    const string outputName = "dataGeneratorTest";
    const string outputPath = "/tmp/";
    const unsigned int pldFormatCode = 1;
    const unsigned int slotNumber = moduleNumber + 2;
    unsigned int runNumber = 0;
    const string runTitle = "Constant Data with Energy = 55, Mod=Chan=Crate=0";
    const unsigned int totalNumberOfWords = 6;

Notes

1. The MVP for this will NOT randomly generate anything. Everything will be static. This provides easier debugging.
2. The data files will be output to /tmp since they can be regenerated at any time and do not need to be stored long term.
3. The program generates both an LDF and a PLD.

From Hui:

​Firmware version 34689 ​(RevF 14-bit, 100 MS/s, no source changes from 34688) should have the same header structure as 34688 (source files changed for all variants). However, ​34455 (RevF 14-bit, 250 MS/s) has slightly different header structure than 34688 and later versions.

34688

​34455

​The change made in 34688 was to accommodate the ​the 16-bit energy values available in the 16-bit variant of RevF.

Right now we've got it setup so that it pulls the repo from scratch. TravisCI does that before we begin the build process :

0.83s$ git clone --depth=50 --branch=dev https://github.com/spaulaus/paass-laughing-conqueror.git spaulaus/paass-laughing-conqueror
Cloning into 'spaulaus/paass-laughing-conqueror'...
remote: Counting objects: 1748, done.
remote: Compressing objects:remote: Compressing objects: 100% (876/876), done.
remote: Total 1748 (delta 941), reused 1342 (delta 824), pack-reused 0
Receiving objects: 100% (1748/1748), 1.15 MiB | 10.55 MiB/s, done.
Resolving deltas: 100% (941/941), done.
$ cd spaulaus/paass-laughing-conqueror
$ git checkout -qf f8d6d2d019b85f19424d0b5f5541dc967c9ca8e2

We do this again in the configuration, but only build off the master branch. This means that dev isn't testing stuff on dev.

Installed the systemd script "plx.service" and ran command "systemctl enable plx", according to Wiki instructions. Rebooted PC, but driver not loaded - I still had to manually call PLX_SDK_DIR=/opt/plx/current/PlxSdk /opt/plx/current/PlxSdk/Bin/Plx_load 9054.

Users with less knowledge of git may get tripped up by the fact that the submodules are not initialized and updated. We can use the following code segment to handle that automatically for them.

#Check if the submodule has been cloned. If not we do it for the user.
set(SUBMODULE_NAME "submodule")
file(GLOB SUBMODULE_FILES "${CMAKE_SOURCE_DIR}/${SUBMODULE_NAME}/*" )
list(LENGTH SUBMODULE_FILES NUM_SUBMODULE_FILES)
if (${NUM_SUBMODULE_FILES} EQUAL 0)
   message(STATUS "Cloning '${SUBMODULE_NAME}' submodule.")
   execute_process(COMMAND git submodule update --init ${CMAKE_SOURCE_DIR}/${SUBMODULE_NAME})
endif (${NUM_SUBMODULE_FILES} EQUAL 0)

The above code segment could be stuck in a loop over all submodules.

An alternate approach is to check for the .git file, but this may cause issues for instances that are downloaded as a zip as they do not contain the git files.

The new method name is much clearer in function.

The minimum slot a pixie module can have is 2. Therefore, we should have error checking in place to ensure that this is never less than 2.

Need to have the CI run unittests for us.

I tried using rootscan (from the dev branch, 42cffe) today to scan some old data and was greeted with this:

$ ./bin/rootscan --frequency=250 --firmware=34688
*** Error in `./bin/rootscan': malloc(): memory corruption (fast): 0x000000000244dd10 ***
Aborted (core dumped)

Running with no arguments gives a similar message.

rootscan doesn't install as there is no CMake instruction for it. Do we want to install this by default or add another option?

Is there ever a situation where we would not require ncurses? I honestly can't think of one.

If we always require it then we can definitely get rid of the CMake flag along with all the if(NCURSES_FOUND) lines.

We don't actually parse out some of the information in the XiaListModeDataDecoder (ex. external TSs ). We should update this so that we can handle all of the different formats of the list mode data.

The firmware is constantly being updated and often we can unpack it using the previous algorithms, but this is not guaranteed. I suggest we warn the user that the unpacking may fail and then attempt to use the old algorithm anyway.

Business Value

If we use this build commend on a system without ROOT installed. The build fails.

cmake ../ -DPAASS_BUILD_UTKSCAN=TRUE -DPAASS_BUILD_TESTS=TRUE

Acceptance Criteria

1. The build succeeds with this scenario.

This is the first time it's happened.

apt-get install failed
$ cat ~/apt-get-update.log
Ign:1 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty InRelease
Hit:2 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty-updates InRelease
Hit:3 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty-backports InRelease
Hit:4 http://us-central1.gce.archive.ubuntu.com/ubuntu trusty Release
Ign:5 http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 InRelease
Hit:6 http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 Release
Hit:7 http://apt.postgresql.org/pub/repos/apt trusty-pgdg InRelease
Hit:8 http://security.ubuntu.com/ubuntu trusty-security InRelease
Ign:9 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:10 http://dl.hhvm.com/ubuntu trusty InRelease
Hit:11 http://dl.google.com/linux/chrome/deb stable Release
Ign:12 http://toolbelt.heroku.com/ubuntu ./ InRelease
Hit:16 http://toolbelt.heroku.com/ubuntu ./ Release
Hit:17 https://download.docker.com/linux/ubuntu trusty InRelease
Get:14 http://dl.bintray.com/apache/cassandra 39x InRelease [3,168 B]
Hit:20 https://packagecloud.io/basho/riak/ubuntu trusty InRelease
Hit:21 https://packagecloud.io/computology/apt-backport/ubuntu trusty InRelease
Hit:22 https://packagecloud.io/github/git-lfs/ubuntu trusty InRelease
Hit:23 https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu trusty InRelease
Err:24 http://ppa.launchpad.net/chris-lea/redis-server/ubuntu trusty InRelease
  Could not connect to ppa.launchpad.net:80 (91.189.95.83), connection timed out
Err:25 http://ppa.launchpad.net/george-edison55/cmake-3.x/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Err:26 http://ppa.launchpad.net/couchdb/stable/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Err:27 http://ppa.launchpad.net/git-core/ppa/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Err:28 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Err:29 http://ppa.launchpad.net/pollinate/ppa/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Err:30 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Err:31 http://ppa.launchpad.net/webupd8team/java/ubuntu trusty InRelease
  Unable to connect to ppa.launchpad.net:http:
Fetched 3,168 B in 10s (316 B/s)
Reading package lists...
W: http://dl.hhvm.com/ubuntu/dists/trusty/InRelease: Signature by key 36AEF64D0207E7EEE352D4875A16E7281BE7A449 uses weak digest algorithm (SHA1)
W: Failed to fetch http://ppa.launchpad.net/chris-lea/redis-server/ubuntu/dists/trusty/InRelease  Could not connect to ppa.launchpad.net:80 (91.189.95.83), connection timed out
W: Failed to fetch http://ppa.launchpad.net/george-edison55/cmake-3.x/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Failed to fetch http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Failed to fetch http://ppa.launchpad.net/git-core/ppa/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Failed to fetch http://ppa.launchpad.net/openjdk-r/ppa/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Failed to fetch http://ppa.launchpad.net/pollinate/ppa/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Failed to fetch http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Failed to fetch http://ppa.launchpad.net/webupd8team/java/ubuntu/dists/trusty/InRelease  Unable to connect to ppa.launchpad.net:http:
W: Some index files failed to download. They have been ignored, or old ones used instead.
The command "sudo -E apt-get -yq --no-install-suggests --no-install-recommends --force-yes install gcc-5 g++-5 gfortran-5 gsl-bin libgsl0-dev lib64ncurses5-dev" failed and exited with 100 during .
Your build has been stopped.

Business Value

TravisCI allows us to test the software against a bunch of different compiler versions and even entire environments. It would be useful if we were to expand the tests to include more variations.

Functional Requirements

1. The tests run against a minimum and maximum compiler version.

Acceptance Criteria

1. All the FRs.
2. All the tests pass.

Developer Notes

1. Not sure how to do this in a clean way other than creating a bunch of build directories and cmake commands with all the permutations.

Introduction

This will serve as a software development log for the PR270 experiment at iThemba LABS. This code is tracked in this branch.

Assumptions

1. The Polarization signals occur when the beam polarization is swapped from Up to Down or Down to Up. I.e. I get a Up logic signal when the beam is switched to up polarizaiton. I get a down polarization signal when the beam is switched to down polarization.
   
2. That the logic signals will be split. This facilitates triggering on both the rising and falling edge of the signal. It provides additional redundancy when recording TTL signals.
3. There will be a pulser signal in the setup to monitor deadtime.
4. We will want to gate the telescopes on beam up or beam down.

The unittests and their headers need some cleaning. Things are working, but I think there's a lot of redundancy in UnitTestSampleData.hpp.

Business Value

This return value is being ignored and the program continues until an exception is thrown. This may be in an issue in some of the utility programs as well. The issue was noticed when using the -h or --help argument from the command line.

Functional Requirements

1. The return value of ScanInterface::Setup is handled so that it doesn't cause future problems in the code.

Not sure where the error is coming from. But I think the mask is a good place to start.

Business Value

If we're not installing with HRIBF dependence then we do not need gfortran. I think we should move the check for gfortran until after we know if the user wants to compile with HRIBF. It removes an optional dependency and allows for smoother testing.

Acceptance Criteria

1. gfortran is not required to build if PAASS_USE_HRIBF=OFF

The wiki information:

does not match what is reflected in HelperEnumerations.cpp:

    ///An enum for the different firmware revisions for the Pixie-16 modules.
    /// * R17562 is valid from 12/15/2010 to 09/26/2011 (this is primarily a
    /// Rev D firmware, i.e. 100 MS/s)
    /// * R20466 is valid from 09/26/2011 to 06/11/2013
    /// * R27361 is valid from 06/11/2013 to 02/15/2014
    /// * R29432 is valid from 02/15/2014 to 07/28/2014
    /// * R30474, R30980, R30981 is valid from 07/28/2014 to 03/08/2016
    /// * R29432 is valid from 03/08/2016
    /// * UNKNOWN is used for unspecified firmware revisions.
    ///These dates do not imply that the particular data set being analyzed was
    /// taken with the expected firmware. These dates are meant only to help
    /// guide the user if they do not know the particular firmware that was used
    /// to obtain their data set.
    enum FIRMWARE {
        R17562, R20466, R27361, R29432, R30474, R30980, R30981, R34688, UNKNOWN
};

or in XiaListModeDataMask.cpp:

    if (firmwareNumber >= 17562 && firmwareNumber < 20466)
        firmware = R17562;
    else if (firmwareNumber >= 20466 && firmwareNumber < 27361)
        firmware = R20466;
    else if (firmwareNumber >= 27361 && firmwareNumber < 29432)
        firmware = R27361;
    else if (firmwareNumber >= 29432 && firmwareNumber < 30474)
        firmware = R29432;
    else if (firmwareNumber >= 30474 && firmwareNumber < 30980)
        firmware = R30474;
    else if (firmwareNumber >= 30980 && firmwareNumber < 30981)
        firmware = R30980;
    else if (firmwareNumber >= 30981 && firmwareNumber < 34688)
        firmware = R30981;
    else if (firmwareNumber ==
             34688) //compare exactly here since nothing higher
        firmware = R34688;
    else {
        msg << "XiaListModeDataMask::CovnertStringToFirmware : "
            << "Unrecognized firmware option - " << type << endl;
        throw invalid_argument(msg.str());
}

Business Value

The CI scripts help ensure that we're not breaking things by accident. We'll need to ensure that they're working.

Acceptance Criteria

1. The TravisCI scripts work properly.

Business Value

Outputting the name of the test before running it allows us to see what tests are being run.

Functional Requirements

1. The bash script running the tests echos the test name to the log before running the test

There's still a define defining the program name. This isn't really that useful....

I suggest we move to using the ExternalProject scheme of CMake for all submodues. This will permit each submodule to have independent CMake projects and remove possible collisions.

Lets take an example situation where you have this setup:

project/
|- CMakeLists.txt
|- external/
   |- CMakeLists.txt
   |- subproject/

There are two ways to do this using git to manage submodules or permitting CMake to handle them. In the latter the subproject directory above would not be present. The main CMakeLists.txt would simply read

add_subdirectory(external)

The external\CMakeLists.txt file would contain

include(ExternalProject)

ExternalProject_Add(subproject
   PREFIX subproject

   #This project can be fetched directly from the git repo
   #GIT_REPOSITORY https://github.com/spaualus/subproject

   #OR if already packaged with submodule we can indicate the location of the source.
   SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/subproject

   CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
)

The above file has selected the former case where the submodule is part of the project as managed by git. To use the latter case the GIT_REPOSITORY line would be uncommented and the SOURCE_DIR line removed. If we want to further restrict where the subproject is being pulled from we could use the GIT_TAG and identify a specific branch, tag, or commit. The submodule would then be cloned by CMake inside the build directory.

More details on CMake's ExternalProject at https://cmake.org/cmake/help/v3.8/module/ExternalProject.html

@jjvs has reported an issue when trying to compile utkscan with all of the flags toggled ON.

The error message

/opt/paass/Analysis/Utkscan/processors/source/CloverProcessor.cpp:101:34: error: ‘class Globals’ has no member named ‘configfile’
     string cfg = Globals::get()->configfile();
                                  ^
/opt/paass/Analysis/Utkscan/processors/source/CloverProcessor.cpp:130:17: error: ‘GeneralException’ was not declared in this scope
 not implemented");
                 ^
/opt/paass/Analysis/Utkscan/processors/source/CloverProcessor.cpp: In member function ‘virtual bool CloverProcessor::Process(RawEvent&)’:
/opt/paass/Analysis/Utkscan/processors/source/CloverProcessor.cpp:712:51: error: ‘class ChanEvent’ has no member named ‘GetCalEnergy’
                         double gEnergy3 = (*it3)->GetCalEnergy();

@jjvz reported having an issue compiling Correlator.

/opt/paass/Analysis/Utkscan/core/source/Correlator.cpp:200:32: error: call of overloaded ‘isnan(double&)’ is ambiguous
             if (!isnan(lastTime)) {

I still think it's weird that these types of issues are caught by some compilers and not caught by others.

Business Value

This could lead to confusion for people reviewing the code.

Functional Requirements

1. Change the brief description to reflect the proper version of GSL.

There are times when the values checked for equality with XiaData are true values. We check that IDs and times are not equal to the defaults of 0. It is possible to have an XiaData event with an ID of 0 and a time of 0.

Business Value

The UTKSCAN flags are legacy and may result in unspecified behavior if they are enabled without knowing the code that they will be enabling. Recommend that they be marked as advanced options for now.

Functional Requirements

1. PAASS_UTKSCAN_* are marked as advanced options

Acceptance Criteria

1. The PAASS_UTKSCAN_* options do not show up in ccmake unless the user says to show advanced.

We'll need to somehow get the PLX library installed along with the xia firmware so that we can test if the DAQ can compile properly.

CLion seems to have an issue formatting the unittests properly due to the macros. We'll need to go through and reformat all of them to be readable.

Description

There is no way to understand the commands supported by RootScanner without reading the source code.

Acceptance Criteria

1. The user is provided a list of commands when typing help

We'll probably want to have a class that handles computing the simple times since the same function can be used for combining the Time Low and Time High.

Business Value

This is pseudo-enforced, but we need it to build utkscan correctly. This is also necessary for building the fitting analyzer.

Functional Requirements

1. GSL is a required package for the software.

cthornsb has written a nice little program to output data files into raw hex. I'm going to roll this into the Analysis utilities.

The exceptions class has a rather crappy implementation. This needs to be updated to reduce the number of lines of code, and provide us a more varied number of exceptions.

When using view baselines I cannot press "Any key to exit" I can only press enter:

Removing lock file /tmp/PixieInterface............[OK]
Press any key to exit...
q

poll2_core.cpp has a number of string manipulation functions starting at line 2094. These should be moved to the StringManipulationFunctions header so that they can be with their buddies. Also unit tested.

Business Value

I'm putting this here mostly to discuss this topic. I think we should remove the HRIBF dependence (pacman mode, USE_HRIBF, utkscanor, etc. This causes an unnecessary amount of clutter for something that we do not intend to use.

Functional Requirements

1. TBD

Business Value

The readme provides some antiquated information. We should update it so that it contains more recent information.

Acceptance Criteria

1. The information in the readme is accurate