Step 1: Install Condor

1.1 Get the zip: wget https://research.cs.wisc.edu/htcondor/tarball/10.x/current/condor-x86_64_CentOS8-stripped.tar.gz

1.2 Untar to your condor folder: tar -x -f condor*.tar.gz mkdir condor cd condor-*stripped mv * ../condor cd .. (rm -rf condor-*stripped rm condor-stripped.tar.gz —> delete the condor.tar.gz and the condor-*stripped)

1.3 Configure: cd condor ./bin/make-personal-from-tarball

Step 2: Install Pegasus

2.1 Get the zip from Tarballs: wget https://download.pegasus.isi.edu/pegasus/5.0.7/pegasus-binary-5.0.7-x86_64_rhel_7.tar.gz

wget https://download.pegasus.isi.edu/pegasus/5.0.7/pegasus-worker-5.0.7-x86_64_rhel_7.tar.gz

2.2. Untar to your pegasus folder (both for pegasus and pegasus-worker): tar zxf pegasus-.tar.gz rm pegasus-.tar.gz

Step 3: Install and compile Montage

3.1 Get the code: git clone https://github.com/Caltech-IPAC/Montage.git

3.2 Compile: make

NOTE: make sure there are no errors. By cloning the GitHub repo we get the most recent version, likely with no compiler errors. cd Montage/bin and make sure it is not empty.

3.3 Save in Paths: export PATH=/usr/workspace/iopp/kogiou1/Montage/bin:$PATH

Step 4: Get the montage-pegasus-v3

4.1 Create and activate Virtual Environment: python3 -m venv /path/to/pegasus-env source /usr/workspace/iopp/kogiou1/venvs/pegasus-env/bin/activate

4.2 Install dependencies: pip install astropy pip install pegasus-wms pip instal "dlio_profiler_py[dlp_analyzer] @ git+https://github.com/hariharan-devarajan/dlio-profiler"

4.3 Get the code: git clone https://github.com/pegasus-isi/montage-workflow-v3.git

Set 5: Compile the pegasus-mpi-cluster from source:

5.1 Get the code: git clone https://github.com/pegasus-isi/pegasus/tree/master

5.2 make sure you’re in the virtual environment for pegasus: source /usr/workspace/iopp/kogiou1/venvs/pegasus-env/bin/activate

5.3 make sure you have the prerequisites:

• Git
• Java 8 or higher
• Python 3.5 or higher
• R
• Ant
• gcc
• g++
• make
• tox 3.14.5 or higher
• mysql (optional, required to access MySQL databases)
• postgresql (optional, required to access PostgreSQL databases)
• Python pyyaml
• Python GitPython

5.4 compile: cd pegasus ant compile-pegasus-mpi-cluster

5.5 copy it to your pegasus folder: cd packages/pegasus-mpi-cluster/ cp pegasus-mpi-cluster/ /usr/workspace/iopp/kogiou1/workflows/pegasus-5.0.7/bin

NOTE: if errors while compiling make sure that MVAPICH is loaded: module load mvapich2-tce/2.3.7 echo $LD_LIBRARY_PATH /usr/tce/packages/mvapich2-tce/mvapich2-2.3.7-intel-classic-2021.6.0/lib:/usr/tce/packages/texlive/texlive-20220321/lib

Step 6: Create a single “install” directory for all pegasus software: (This will help in resolving errors like “cannot find .. in your path”)

6.1 Move into pegasus directory (the one you compiled from source) and make a directory called install: cd pegasus mkdir install

6.2 Copy all components from pegasus-5.0.7 and condor in the pegasus/install folder: cd ../condor cp * ../pegasus/install cp -r * ../pegasus/install cd ../pegasus-5.0.7 cp * ../pegasus/install cp -r * ../pegasus/install

NOTE: if error for overwriting /bin or /lib folders you have to do it manually by cd inside those folders and cp everything to /pegasus/install/bin or /pegasus/install/lib. Make sure all the components are there otherwise pegasus and condor cannot run

Step 7: Run workflows with pegasus: 7.1 Allocate a node with flux: flux alloc -N3 -n48

NOTE: make sure you are in the virtual environment still. If not source it again by repeating 5.2

7.2 Save to PATH: export PATH=/usr/workspace/iopp/kogiou1/workflows/pegasus/install/bin:$PATH export PATH=/usr/workspace/iopp/kogiou1/workflows/pegasus/install/sbin:$PATH export LD_LIBRARY_PATH=/usr/workspace/iopp/kogiou1/workflows/pegasus/install//lib:$LD_LIBRARY_PATH (source ~/.bashrc)

7.3 Run Condor: chmod 777 /usr/workspace/iopp/kogiou1/workflows/pegasus/install/condor.sh . /usr/workspace/iopp/kogiou1/workflows/pegasus/install/condor.sh condor_master condor_status (it should show the activity) condor_q (it should show the jobs running)

NOTE: if errors echo the LD_LIBRARY_PATH and the PATH and make sure /pegasus/install is there

NOTE: To check if condor_shedd and all other condor processes are running ps aux | grep condor

NOTE: if condor throws error while trying to connect to another node then: Exit the flux allocation: exit Check your processes: ps -u USER Kill all your processes (or those related to condor if any): killall -u USER Repeat 6.3, 6.4, 5.2, 6.5 If again problem: condor_restart, 6.5

7.4 Test Pegasus: pegasus-version (should show 5.0.7)

NOTE: If Error: Cannot find file with permissions: touch that file and make sure it has those permissions

7.5 Configure the Condor/SLURM interface pegasus-configure-glite

NOTE: If Error: Cannot find file with permissions: touch that file and make sure it has those permissions

flux batch -N16 -n768 -c1 --queue=pbatch -t 30m --setattr=system.bank=asccasc ./run_corona.sh

flux batch -N32 -n1536 -c1 --queue=pbatch -t 360m --setattr=system.bank=asccasc ./run_corona.sh

To find if thw job is running: flux jobs To find if there has been an error for the workflow: pegasus-analyzer $PWD