High Performance Computing
The following document will aid an administrator on setting up an environment which end-users will need when building and running MOOSE based applications. If you do not have administrative rights, you will not be able to complete these instructions. Please forward these instructions to your HPC administrator.
Prerequisites
Some sort of environmental module management software. Such as Environment Modules.
A working MPI wrapper (MPICH/OpenMPI/MVAPICH) which wraps to a C++11 compliant compiler.
Knowledge on how to set up the environment to use the desired compiler (either module loading, or exporting PATHs etc)
Please use a single solitary terminal session throughout and to the completion of these instructions.
Environment Setup
Begin by creating an area for which to build:
export STACK_SRC=`mktemp -d /tmp/stack_src_temp.XXXXXX`
cd $STACK_SRC
Set your umask
Some systems have a secure umask set. We need to adjust our umask so that when you write a file, it is readable by everyone:
umask 0022
Choose a base path
Export a base path variable which will be the home location for the compiler stack. All files related to libraries necessary to build MOOSE, will be stored in this location (choose carefully, as this location should be accessible from all nodes on your cluster):
export PACKAGES_DIR=/opt/moose
Create and chown $PACKGES_DIR
History teaches us, that implicitly trusting scripts we download off the internet with root access, is a very bad idea. So let us create and chown the $PACKAGES_DIR directory before we install anything. That way, the things we do install can be done so without invoking sudo:
sudo mkdir $PACKAGES_DIR
sudo chown -R <your user id> $PACKAGES_DIR
Verify that your umask settings are indeed set to 0022 before continuing:
$> umask
0022
Set Up Modules
Even if you are not using Modules, the following provides information on what environment variables are needed for MOOSE developement.
Create a MOOSE module:
mkdir -p $PACKAGES_DIR/modulefiles
vi $PACKAGES_DIR/modulefiles/moose-dev-gcc
Add the following content to that file:
#%Module1.0#####################################################################
##
## MOOSE module
##
set base_path INSERT PACKAGES_DIR HERE
GCC MPI PATHS
prepend-path PATH /GCC and MPI /bin
prepend-path LD_LIBRARY_PATH /GCC and MPI /lib
setenv CC mpicc
setenv CXX mpicxx
setenv F90 mpif90
setenv F77 mpif77
setenv FC mpif90
setenv PETSC_DIR $base_path/petsc/petsc-3.8.3/gcc-opt
# Optional if miniconda is installed
prepend-path PATH $base_path/miniconda/bin
Replace INSERT PACKAGES_DIR HERE with whatever you had set for $PACKAGES_DIR (do not literally enter: $PACKAGES_DIR. As an example, if you left packages_dir as: /opt/moose, then that is what you would enter)
Replace GCC and MPI paths with any additional information needed to make GCC/MPI work.
To make the module available in your terminal session, export the following:
export MODULEPATH=$MODULEPATH:$PACKAGES_DIR/modulefiles
The above command should be added to the list of other global profiles (perhaps in /etc/profiles.d). That way, the above is performed as the user logs into the machine.
With the modulefile in place and the MODULEPATH variable set, see if our module is available for loading:
module load moose-dev-gcc
Verify that this module loads properly by attempting to echo $PETSC_DIR:
echo $PETSC_DIR
/opt/moose/petsc/petsc-3.8.3/gcc-opt
While we are at it, verify that your MPI wrapper works by running a few commands (your results will vary, but they should return something):
which mpicc
/opt/moose/mpich-3.2/gcc-7.3.0/bin/mpicc
mpicc -show
gcc -I/opt/moose/mpich-3.2/gcc-7.3.0/include -L/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,-rpath -Wl,/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,--enable-new-dtags -lmpi
which gcc
/opt/moose/gcc-7.3.0/bin/gcc
Leave this module loaded for the remainder of the instructions (PETSc requirements).
PETSc
Download PETSc 3.8.3
cd $STACK_SRC
curl -L -O http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.3.tar.gz
tar -xf petsc-3.8.3.tar.gz -C .Now we configure, build, and install it
cd $STACK_SRC/petsc-3.8.3
./configure \
--prefix=$PACKAGES_DIR/petsc-3.8.3 \
--download-hypre=1 \
--with-ssl=0 \
--with-debugging=no \
--with-pic=1 \
--with-shared-libraries=1 \
--with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-mumps=1 \
--download-scalapack=1 \
--CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 \
--CFLAGS='-fPIC -fopenmp' \
--CXXFLAGS='-fPIC -fopenmp' \
--FFLAGS='-fPIC -fopenmp' \
--FCFLAGS='-fPIC -fopenmp' \
--F90FLAGS='-fPIC -fopenmp' \
--F77FLAGS='-fPIC -fopenmp' \
PETSC_DIR=`pwd`
Once configure is done, we build PETSc
make PETSC_DIR=$STACK_SRC/petsc-3.8.3 PETSC_ARCH=arch-linux2-c-opt allEverything good so far? PETSc should be asking to run more make commands
make PETSC_DIR=$STACK_SRC/petsc-3.8.3 PETSC_ARCH=arch-linux2-c-opt installAnd now after the install, we can run some built-in tests
make PETSC_DIR=$PACKAGES_DIR/petsc-3.8.3 PETSC_ARCH="" testRunning the tests should produce some output like the following:
[moose@centos-7 petsc-3.8.3]$ make PETSC_DIR=$PACKAGES_DIR/petsc-3.8.3 PETSC_ARCH="" test
Running test examples to verify correct installation
Using PETSC_DIR=/opt/moose/petsc-3.8.3 and PETSC_ARCH=
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
=========================================Miniconda
Peacock (an optional MOOSE GUI frontend) uses many libraries. The easiest way to obtain these libraries, is to install miniconda, along with several miniconda/pip packages.
cd $STACK_SRC
curl -L -O https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
sh Miniconda2-latest-Linux-x86_64.sh -b -p $PACKAGES_DIR/miniconda
PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda config --set ssl_verify false
PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda install -c idaholab python=2.7 coverage \
reportlab \
mako \
numpy \
scipy \
scikit-learn \
h5py \
hdf5 \
scikit-image \
requests \
vtk=7.1.0 \
pyyaml \
matplotlib \
pip \
lxml \
pyflakes \
pandas \
conda-build \
mock \
yaml \
pyqt \
swig --yes
Peacock (as well as the TestHarness sytem in MOOSE), does not work with Python3. Please chose Miniconda2 for Python 2.7 instead.
Next, we need to use pip to install additional libraries not supplied by conda:
PATH=$PACKAGES_DIR/miniconda/bin:$PATH pip install --no-cache-dir pybtex livereload==2.5.1 daemonlite pylint==1.6.5 lxml pylatexenc anytree
Clean Up and Chown
Clean all the temporary stuff and change the ownership to root, so no further writes are possible:
rm -rf $STACK_SRC
sudo chown root $PACKAGE_DIR
This concludes setting up the environment for MOOSE-based development. However you decide to instruct your users on enabling the above environment, each user will need to perform the instructions provided by the following link: Obtaining and Building MOOSE.