Installing and Running on Ubuntu

Users interested to work with PFLOTRAN-OGS at code level, can proceed with the installation and build from source code.

Ubuntu verions 16.04 and 18.04 are supported but not currently 20.04.

Below instructions are provided to install the software on the Ubuntu Desktop system. Support for clients working with their own builds is arranged on a case-by-case basis. Please get in touch with OpenGoSim to know more.

Requirements

We assume an up to date Ubuntu 16.04 or 18.04 installation, as well as:

The software managment tool git, which can be installed with:

sudo apt install git

Compilers for Fortran, C, C++, and other necesasary compilation tools and librares:

sudo apt install build-essential

Flex and Bison are also required:

sudo apt install flex
sudo apt install bison

The other major requirment for PFLOTRAN-OGS is the library PETSc, which we will install in the next section. PETSc has a number of other dependencies we will need, but fortunately it can be configured to download and install all of these automatically, see the next section.

Downloading PFLOTRAN-OGS

Use git to clone the PFLOTRAN-OGS.

For OGS-1.5:

git clone https://bitbucket.org/opengosim/pflotran_ogs_1.5.git

This will place PFLOTRAN-OGS in a folder called pflotran-ogs-1.5.

For OGS-1.4 please use:

git clone https://bitbucket.org/opengosim/pflotran_ogs_1.4.git

This will place PFLOTRAN-OGS in a folder called pflotran_ogs_1.4.

Installing PETSc

Now we will download and install the PETSc library.

Create a directory for PETSc, for example:

mkdir petsc

Clone the PETSc repository from git into that directory using the following command:

git clone https://gitlab.com/petsc/petsc petsc

For compiling PETSc and PFLOTRAN-OGS we have to set some simple environmental variables. This is further explained in PETSc environmental variables but is not required reading.

First we must set PETSC_DIR to the path of the directory we cloned PETSc into, so for example:

export PETSC_DIR=/home/myusername/petsc

We must also set a variable called PETSC_ARCH. This is used for managing multiple installs/configurations of PETSc, so we could use anything vaguely descriptive. Following some standard convention let’s use:

export PETSC_ARCH=ubuntu-opt

because we’re installing on Ubuntu, and we’re going to configure for a fast running (optimised ‘opt’, as opposed to debugging, ‘dbg’) version of PETSc.

We’re going to need these variables defined every time we do any compilation of PETSc or PFLOTRAN, so it might be convinient to also paste these two lines into the .bashrc file so that they are defined every time a new terminal window is opened.

PFLOTRAN-OGS needs a specific snapshot of PETSc. We can check what this is in the file petsc-git-version.txt, included in the copy of PFLOTRAN-OGS we downloaded. For example:

nano pflotran_ogs_1.5/tools/buildbot/petsc/petsc-git-version.txt

For example the version at time of writing is v3.12.2, which we will use in this tutorial.

Next we go into the PETSc directory:

cd petsc

And checkout the specific snapshot which PFLOTRAN-OGS needs, for example:

git checkout v3.12.2

Now we configure PETSc. Among other things, this is the part where PETSc will download and install any dependent libraries if we ask it to. So we will ask it to:

./configure --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes  --download-cmake=yes  --download-ptscotch=yes download-hypre=yes --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3

If you see an error here along the lines of:

/usr/bin/env ‘python’ No such file or directory

this is likely due to Python 2 not being installed on your Ubuntu version. It can be installed with:

sudo apt install python

Another possible, rare error is that gfortran was not installed by build-essential. If you see error messages relating to Fortran, this might be the case and can be fixed with:

sudo apt install gfortran

For advanced user information about compiling PETSc against existing installs on any of these libraries, for example mpich or replacing it with openmpi, see this section.

The configure process will take a litle time. Eventually the configuration script will finish, and even tell us how to proceed. We should see and output like this at the end:

xxx=========================================================================xxx
 Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt all
xxx=========================================================================xxx

So we copy paste this into the terminal:

make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt all

Then we wait again for the build to complete. Eventually we will see more advice from the scripts:

Now to check if the libraries are working do:
make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt check
=========================================

We again oblige by copy pasting:

make PETSC_DIR=/home/myusername/petsc PETSC_ARCH=ubuntu-opt check

Giving an output something like:

ubuntu-opt check
Running test examples to verify correct installation
Using PETSC_DIR=/home/daniel/new_petsc and PETSC_ARCH=ubuntu-opt
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/examples/tutorials/ex19 run successfully with hypre
C/C++ example src/vec/vec/examples/tutorials/ex47 run successfully with hdf5
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples

Installing PFLOTRAN-OGS

Now PETSc should be installed correctly and we are ready to move on to PFLOTRAN-OGS.

We have already downloaded PFLOTRAN-OGS through git, and have it in a folder called pflotran_ogs_1.5.

Navigate into the following directory:

cd pflotran_ogs_1.5/src/pflotran

Now we compile (in this case with four cores, to speed things up) by entering:

make -j4 pflotran

This should take a few minutes, after which, PFLOTRAN-OGS will be ready to use.

Running Pflotran From the Command Line

This section will help you understand the options available for running PFLOTRAN-OGS from the command line.

Through Scripts

In practice it is much more convinient to run PFLOTRAN-OGS through simple scripts. We provide an example here. Please open the file and read the short instructions before running. In particualar note that it must be updated with the path of your pflotran install directory (e.g., /home/myusername/pflotran), and needs the PETSC_DIR and PETSC_ARCH variables to be defined (see above).

Directly Through the Command Line

A typical call to run PFLOTRAN-OGS from the command line will look like

/home/myusername/petsc/ubuntu-opt/bin/mpirun -np 4 /home/myusername/pflotran/src/pflotran -pflotranin spe10.in -output_prefix test_spe10_run

We now explain each part of this:

  • /home/myusername/petsc/ubuntu-opt/bin/mpirun : we call mpirun to start a parallel program. Note that this is the mpirun binary that was installed as part of the PETSc configuration process above. We need to make sure we use the same MPI installation for compiling and running PFLOTRAN-OGS.

  • -np 4 : an argument to mpirun, specifying how many proceseses to use. In this case, four.

  • /home/myusername/pflotran/src/pflotran : this is the PFLOTRAN-OGS binary we compiled earlier.

  • -pflotranin spe10.in : this is an argument to PFLOTRAN-OGS, specifying an input file to use, in this case called ‘spe10.in’.

  • -output_prefix test_spe10_run : specify an output prefix to be applied to all output files generated by this run.

Advanced PETSc Options

Configuring against existing Open MPI Install

Here is an example:

CONFIGURE_OPTIONS = --with-debugging=0 --download-fblaslapack=1 --with-fc=/usr/lib64/openmpi/bin/mpif90 --with-cc=/usr/lib64/openmpi/bin/mpicc --with-cxx=/usr/lib64/openmpi/bin/mpicxx --with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-lib=/usr/lib64/openmpi/lib/libmpi.so --download-cmake=1 --download-ptscotch=1 -download-hypre=1 --download-hdf5=1 --with-c2html=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-shared-libraries=0

Note the lack of the --download-mpich=yes or --download-openmpi=yes options, as well as the presence of the following:

  • A path to an MPI fortran compiler: --with-fc=/usr/lib64/openmpi/bin/mpif90

  • A path to an MPI c compiler: --with-cc=/usr/lib64/openmpi/bin/mpicc

  • A path to an MPI c++ compiler: --with-cxx=/usr/lib64/openmpi/bin/mpicxx

  • A path to to the MPI include directory: --with-mpi-include=/usr/include/openmpi-x86_64

  • A path to the MPI libraries: --with-mpi-lib=/usr/lib64/openmpi/lib/libmpi.so

Note that it might be necessary to ensure certain libraries are in path, e.g.

export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH

Recall that you must use the mpirun binary associted with the Open MPI install for running PFLOTRAN-OGS, so the above example of running on the command line will generalizes to:

/location/of/correct/mpirun -np 4 /home/myusername/pflotran/src/pflotran -pflotranin spe10.in -output_prefix test_spe10_run