Installing and Running on Ubuntu
Users interested to work with Cirrus at code level can build and install the application from source code.
Ubuntu versions 24.04 and 22.04 are supported.
Below instructions are provided to install the software on the Ubuntu Desktop system. We also show an example of a simple script for automating most of this process.
Support for clients working with their own builds is arranged on a case-by-case basis. Please get in touch with OpenGoSim to know more.
Warning: certain environmental variables set by scripts included with Intel OneApi can conflict with the build process. In particular, please ensure that the script setvars.sh bundled with OneApi has not been run in the environment used for the following build. If you don’t have OneApi installed on your Ubuntu system, this warning can be ignored.
Manual Install on Ubuntu 24.04
Requirements
We assume an up to date Ubuntu 24.04 installation. The following packages from apt are also required. If you are interested in understanding why each package is needed, we describe them individually below. Otherwise, the following command to install all at once should be sufficient:
sudo apt install git build-essential gfortran python3 python3-six flex bison
Looking at these individually:
git: The software managment tool git will be needed to download Cirrus and PETSc (see below).
build-essential and gfortran provide compilers for Fortran, C, C++, and other necesasary compilation tools and librares.
python3 and python3-six: Python, and this specific module, are required for several install scripts.
Flex and Bison are required by PETSc (see below).
The other major requirment for Cirrus is the library PETSc, which we will install in the next section. PETSc has a number of other dependencies we will need, but fortunately it can be configured to download and install all of these automatically, see the next section.
Downloading Cirrus
The latest public version of Cirrus is kept in a public Bitbucket repo. Clone the release branch as follows:
git clone -b release git@bitbucket.org:opengosim/cirrus_1.9.git
This will place Cirrus in a folder called cirrus_1.9.
Downloading PETSc
Now we will download and install the PETSc library. We use a specific version of PETSc - 3.21.5.
Clone the PETSc repository from git into that directory using the following command:
git clone https://gitlab.com/petsc/petsc.git petsc
Go into the petsc directory and checkout the branch for version 3.21.5:
cd petsc
git checkout v3.21.5
Configuring PETSc
For compiling PETSc and Cirrus we have to set some simple environmental variables. This is further explained in PETSc environmental variables but is not required reading.
First we must set PETSC_DIR to the path of the directory we cloned PETSc into, so for example:
export PETSC_DIR=/home/myusername/petsc
We must also set a variable called PETSC_ARCH. This is used for managing multiple installs/configurations of PETSc, so we could use anything vaguely descriptive. Following some standard convention let’s use:
export PETSC_ARCH=ubuntu-opt
because we’re installing on Ubuntu, and we’re going to configure for a fast running (optimised ‘opt’, as opposed to debugging, ‘dbg’) version of PETSc.
We’re going to need these variables defined every time we do any compilation of PETSc or Cirrus, so it might be convinient to also paste these two lines into the .bashrc file so that they are defined every time a new terminal window is opened.
Next we go into the PETSc directory:
cd petsc
Now we configure PETSc. Among other things, this is the part where PETSc will download and install any dependent libraries if we ask it to.
We configure with the following command:
./configure --download-mpich=yes --download-hdf5=yes --with-hdf5-fortran-bindings=yes --download-fblaslapack=yes --download-ptscotch=yes --download-hypre=yes --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
For advanced user information about compiling PETSc against existing installs on any of these libraries, for example mpich or replacing it with openmpi, see this section.
Installing PETSc
The configure process will take a litle time. Eventually the configuration script will finish, and even tell us how to proceed. We should see and output like this at the end:
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt all
xxx=========================================================================xxx
So we copy paste this into the terminal:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt all
Then we wait again for the build to complete. Eventually we will see more advice from the scripts:
Now to check if the libraries are working do:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt check
=========================================
We again oblige by copy pasting:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt check
Giving an output something like:
ubuntu-opt check
Running test examples to verify correct installation
Using PETSC_DIR=/home/daniel/new_petsc and PETSC_ARCH=ubuntu-opt
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/examples/tutorials/ex19 run successfully with hypre
C/C++ example src/vec/vec/examples/tutorials/ex47 run successfully with hdf5
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
Installing Cirrus
Now PETSc should be installed correctly and we are ready to move on to Cirrus.
We have already downloaded Cirrus, and have it in a folder called cirrus_1.9.
Navigate into the following directory:
cd cirrus_1.9/src/cirrus
Now we compile (in this case with four cores, to speed things up) by entering:
make -j4 cirrus
This should take a few minutes, after which, Cirrus will be ready to use.
Install Script
The following is an example of a simple bash script encapsulates the above steps:
#!/bin/bash
export PETSC_DIR=/home/myusername/petsc
export PETSC_ARCH=ubuntu-opt
git clone https://gitlab.com/petsc/petsc.git $PETSC_DIR
cd $PETSC_DIR
git checkout v3.21.5
./configure --download-mpich=yes --download-hdf5=yes --with-hdf5-fortran-bindings=yes --download-fblaslapack=yes --download-ptscotch=yes --download-hypre=yes --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
make
make check
cd ..
git clone -b release git@bitbucket.org:opengosim/cirrus_1.9.git
cd cirrus_1.9/src/cirrus
make -j4 cirrus
Don’t forget to change the PETSC_DIR definition to something sensible (it should be a directory in your home directory where you want to put the PETSc library). This covers downloading and building PETSc and then Cirrus. It is also important to ensure the prerequisites are installed before runing the script, and validate the install after.
Manual Install on Ubuntu 22.04
Requirements
We assume an up to date Ubuntu 22.04 installation. The following packages from apt are also required. If you are interested in understanding why each package is needed, we describe them individually below. Otherwise, the following command to install all at once should be sufficient:
sudo apt install git build-essential gfortran python3 python3-six python-six flex bison
Looking at these individually:
git: The software managment tool git will be needed to download Cirrus and PETSc (see below).
build-essential and gfortran provide compilers for Fortran, C, C++, and other necesasary compilation tools and librares.
python3 and python3-six: Python, and this specific module, are required for several install scripts.
Flex and Bison are required by PETSc (see below).
The other major requirment for Cirrus is the library PETSc, which we will install in the next section. PETSc has a number of other dependencies we will need, but fortunately it can be configured to download and install all of these automatically, see the next section.
Downloading Cirrus
The latest public version of Cirrus is kept in a public Bitbucket repo. Clone the release branch as follows:
git clone -b release git@bitbucket.org:opengosim/cirrus_1.9.git
This will place Cirrus in a folder called cirrus_1.9.
Downloading PETSc
Now we will download and install the PETSc library. We use a specific version of PETSc - 3.19.1.
Clone the PETSc repository from git into that directory using the following command:
git clone https://gitlab.com/petsc/petsc.git petsc
Go into the petsc directory and checkout the branch for version 3.19.1:
cd petsc
git checkout v3.19.1
Configuring PETSc
For compiling PETSc and Cirrus we have to set some simple environmental variables. This is further explained in PETSc environmental variables but is not required reading.
First we must set PETSC_DIR to the path of the directory we cloned PETSc into, so for example:
export PETSC_DIR=/home/myusername/petsc
We must also set a variable called PETSC_ARCH. This is used for managing multiple installs/configurations of PETSc, so we could use anything vaguely descriptive. Following some standard convention let’s use:
export PETSC_ARCH=ubuntu-opt
because we’re installing on Ubuntu, and we’re going to configure for a fast running (optimised ‘opt’, as opposed to debugging, ‘dbg’) version of PETSc.
We’re going to need these variables defined every time we do any compilation of PETSc or Cirrus, so it might be convinient to also paste these two lines into the .bashrc file so that they are defined every time a new terminal window is opened.
Next we go into the PETSc directory:
cd petsc
Now we configure PETSc. Among other things, this is the part where PETSc will download and install any dependent libraries if we ask it to.
We configure with the following command:
./configure --download-mpich=yes --download-hdf5=yes --with-hdf5-fortran-bindings=yes --download-fblaslapack=yes --download-ptscotch=yes --download-hypre=yes --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
For advanced user information about compiling PETSc against existing installs on any of these libraries, for example mpich or replacing it with openmpi, see this section.
Installing PETSc
The configure process will take a litle time. Eventually the configuration script will finish, and even tell us how to proceed. We should see and output like this at the end:
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt all
xxx=========================================================================xxx
So we copy paste this into the terminal:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt all
Then we wait again for the build to complete. Eventually we will see more advice from the scripts:
Now to check if the libraries are working do:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt check
=========================================
We again oblige by copy pasting:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt check
Giving an output something like:
ubuntu-opt check
Running test examples to verify correct installation
Using PETSC_DIR=/home/daniel/new_petsc and PETSC_ARCH=ubuntu-opt
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/examples/tutorials/ex19 run successfully with hypre
C/C++ example src/vec/vec/examples/tutorials/ex47 run successfully with hdf5
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
Installing Cirrus
Now PETSc should be installed correctly and we are ready to move on to Cirrus.
We have already downloaded Cirrus, and have it in a folder called cirrus_1.9.
Navigate into the following directory:
cd cirrus_1.9/src/cirrus
Now we compile (in this case with four cores, to speed things up) by entering:
make -j4 cirrus
This should take a few minutes, after which, Cirrus will be ready to use.
Install Script
The following is an example of a simple bash script encapsulates the above steps:
#!/bin/bash
export PETSC_DIR=/home/myusername/petsc
export PETSC_ARCH=ubuntu-opt
git clone https://gitlab.com/petsc/petsc.git $PETSC_DIR
cd $PETSC_DIR
git checkout v3.19.1
./configure --download-mpich=yes --download-hdf5=yes --with-hdf5-fortran-bindings=yes --download-fblaslapack=yes --download-ptscotch=yes --download-hypre=yes --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
make
make check
cd ..
git clone -b release git@bitbucket.org:opengosim/cirrus_1.9.git
cd cirrus_1.9/src/cirrus
make -j4 cirrus
Don’t forget to change the PETSC_DIR definition to something sensible (it should be a directory in your home directory where you want to put the PETSc library). This covers downloading and building PETSc and then Cirrus. It is also important to ensure the prerequisites are installed before runing the script, and validate the install after.
Running Cirrus
Cirrus can be run directy from the command line, through a utility script provided with the repo, or through our frontend Stratus.
More information about using the script, and getting started with Cirrus in general, can be found here.
More information about running directly through the command line can be found here.
Testing the Cirrus Installation - Regression Tests
A set of fast automatic regression tests can be run to check that the Cirrus installation is working correctly.
To do this, after installing in the previous section, in the same directory, enter:
make test
This will run the tests. First a number of unit tests will be run, to check individual parts of the code, then the regression tests will be run, to check the overall simulator.
When running the regression tests, the screen will look like this:
Running cirrus regression tests :
Legend
. - success
F - failed regression test (results are outside error tolerances)
M - failed regression test (results are FAR outside error tolerances)
G - general error
U - user error
X - code crashed
T - time out error
C - configuration file [.cfg] error
I - missing information (e.g. missing files)
A - pre-processing error (e.g. error in simulation setup scripts
B - post-processing error (e.g. error in solution comparison)
S - test skipped
W - warning
? - unknown
.............................................................................................................................................................................................
--------------------------------------------------------------------------------
Regression test summary:
Total run time: 88.5747 [s]
Total tests : 189
Tests run : 189
All tests passed.
The regression tests are simple simulation models that can be completed very quickly. The values of various physical properties (e.g. pressure, gas saturation, etc.) in certain cells are output to a .regression file at the end of the run. The regression test script compares these values to the same values from an earlier run, deemed to be correct (called “gold” values), and alerts us if the values differ by a nontrivial amount.
Thus the regression tests tell us if the physical properties computed by the simulator have changed since the “gold” values were defined.
After running the regression tests, we should see all dots, indicating all successful tests.
It is possible for some regression tests to fail simply due to slight changes in system/compiler setup causing changes in, for example, rounding error, causing the final physical values to deviate from the “gold” values by slightly more than the specified tolerances, which are quite strict.
In this case the above output will show some “F”s instead of “.”s, indicating the failures. Up to a few, say five, failures are likely due to be due to minor system variations as just described, but many more or all “F”s indicate something has gone wrong.
Another possible outcome is that some tests will show “T”, indicating that they have exceeded 60 seconds in runtime, which is considrered a failure. This can be the case when an outdated machine is used. This can also happen when the machine has few cores, and or OpenMPI has not been setup correctly (note that in the PETSc install above we allowed PETSc to mpich itself, but we may not always choose to do that, see later sections).
Other outputs indicate more serious problems. Seeing “M” indicates that a regression test has failed by producing output very different from what was expected, which cannot be explained by differences between systems. Seeing “G”, “U” or “X” indicates that the simulator has thrown an error or outright crashed. In all these cases the install has not been successful.
To take a closer look at the results of the tests, a full log is stored in:
[cirrus directory]/regression_tests/cirrus-tests-[time tests were run].testlog
An example might be:
[home directory]/cirrus_ogs_1.9/regression_tests/cirrus-tests-2021-06-23_12-12-44.testlog
In this log file we can see details of all the tests that were run. Most importantly, if a test has failed, we can see additional information. If a test failed with “F” or “M”, we can see the deviation between physical values and the “gold values. An example of an “F” is shown below:
cp_np2...
Run...
cd /shared/cirrus/regression_tests/towg/bo
/opt/intel/compilers_and_libraries_2019/linux/mpi/intel64/bin/mpiexec -np 2 /shared/cirrus/src/cirrus/cirrus -malloc_debug no -successful_exit_code 86 -input_prefix cp_np2
# cp_np2 : run time : 0.40 seconds
diff cp_np2.regression.gold cp_np2.regression
FAIL: Liquid Energy:Min : 1.90003568434e-12 > 1e-12 [absolute]
FAIL: Liquid Energy:1 : 2.19979590099e-12 > 1e-12 [absolute]
FAIL: Liquid Energy:Mean : 4.49995596341e-12 > 1e-12 [absolute]
FAIL: Oil Density:Min : 5.50016920897e-10 > 1e-12 [absolute]
FAIL: Oil Density:300 : 5.50016920897e-10 > 1e-12 [absolute]
FAIL: Oil Density:Max : 1.5500063455e-09 > 1e-12 [absolute]
FAIL: Oil Density:1 : 1.8300170268e-09 > 1e-12 [absolute]
FAIL: Oil Density:Mean : 3.73995590053e-09 > 1e-12 [absolute]
FAIL: Oil Energy:Min : 2.34499974994e-10 > 1e-12 [absolute]
FAIL: Oil Energy:300 : 7.69997399175e-11 > 1e-12 [absolute]
FAIL: Oil Energy:Max : 7.69997399175e-11 > 1e-12 [absolute]
FAIL: Oil Energy:1 : 2.72299960358e-10 > 1e-12 [absolute]
FAIL: Oil Energy:Mean : 5.42000000436e-10 > 1e-12 [absolute]
FAIL: Gas Density:Min : 3.73000830223e-09 > 1e-12 [absolute]
FAIL: Gas Density:300 : 3.73000830223e-09 > 1e-12 [absolute]
FAIL: Gas Density:Max : 8.8799652076e-09 > 1e-12 [absolute]
FAIL: Gas Density:1 : 1.24500161292e-08 > 1e-12 [absolute]
FAIL: Gas Density:Mean : 2.50300047355e-08 > 1e-12 [absolute]
FAIL: Gas Energy:Min : 1.05995212607e-11 > 1e-12 [absolute]
FAIL: Gas Energy:Mean : 2.50022225146e-12 > 1e-12 [absolute]
Skipping SOLUTION : Flow
cp_np2... failed.
For example, the average gas density has devaiated from the expected by about 1.0e-8, which is not much, but this is greater than the 1.0e-12 tolerance expected. It is always worth double checking that regression test failures are of this trivial sort when running the regression tests.
In the case of “G”, “U” or “X” failures, the output from the simulator can be found in the log file, often allowing us to see error messages useful for debugging.
After running the tests, if everything went fine, we can clean up, which includes deleting the log files, with:
make clean-tests
It is good practice to run make clean-tests between runs of make test. In particular this allows the unit tests to be rebuilt cleanly and correctly. A handy one-liner when re-running the tests is of course:
make clean-tests && make test
Advanced PETSc Options
Configuring against existing Open MPI Install
On an established HPC system, we might want to use an existing install of of MPI (e.g. OpenMPI, MPICH) instead of having PETSc download and build one. For this we can change the configuration options given to PETSc in this step.
Here is an example:
./configure --with-debugging=0 --download-fblaslapack=1 --with-fc=/usr/lib64/openmpi/bin/mpif90 --with-cc=/usr/lib64/openmpi/bin/mpicc --with-cxx=/usr/lib64/openmpi/bin/mpicxx --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-lib=/usr/lib64/openmpi/lib/libmpi.so --download-cmake=1 --download-ptscotch=1 -download-hypre=1 --download-hdf5=1 --with-c2html=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-shared-libraries=0
Note the lack of the --download-mpich=yes or --download-openmpi=yes options, as well as the presence of the following:
A path to an MPI fortran compiler:
--with-fc=/usr/lib64/openmpi/bin/mpif90A path to an MPI c compiler:
--with-cc=/usr/lib64/openmpi/bin/mpiccA path to an MPI c++ compiler:
--with-cxx=/usr/lib64/openmpi/bin/mpicxxA path to to the MPI include directory:
--with-mpi-include=/usr/include/openmpi-x86_64A path to the MPI libraries:
--with-mpi-lib=/usr/lib64/openmpi/lib/libmpi.so
In this case, we assume a functioning Open MPI build on our system located at:
/usr/lib64/openmpi/
Note that it might be necessary to ensure certain libraries are in path, e.g.
export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH
Recall that you must use the mpirun binary associted with the Open MPI install for running Cirrus, so the above example of running on the command line will generalizes to:
/location/of/correct/mpirun -np 4 /home/myusername/cirrus/src/cirrus -cirrusin spe10.in -output_prefix test_spe10_run
This can also be done with other packages, if they are already present on your system, such as Hypre, HDF5, etc. To find the appropriate flags to pass to PETSc’s configure script, use:
./configure --help
This produces quite a lot of output. It may be convinent to capture this in a text file:
./configure --help > confhelp.txt
which can then be searched for information about the required packages.
Note that this process may cause problems if the package on your system is of a version not compatible with PETSC 3.19.1. This version of PETSc would download and build OpenMPI 4.5.1 for example, so it is reccomended to have this version of later on your system.