Manual Environment Setup on OSC (Pitzer)
- Note
- You should probably be using Cardinal, see Optional: Cardinal
This is modified from the guide written by William Luszczak.
- Tested on CentOS 7, OSC, Midway2, Midway3.
- However, if you are on Midway, you should probably just load the modules (see if you are on Midway 3)
- No
sudo access required
- In this guide, we avoid using
module load and instead opt to compile any dependencies from source. You might be able to get this to work by module loading, for example, fftw, but try this at your own peril.
Prerequisites
Listed below are what Will and Jason are using.
- Python 3.9.18 (OSC & Midway; Jason)
- Note
- needed for PYROOT
cmake 3.21.2 (I had problems with 3.11.4; OSC; Will)
cmake 3.28.4 (Apr. 6 2024; OSC; Jason)
cmake 3.31.1 (Nov. 28 2024; Midway; Jason)
gcc 11.1.0 (9.X will not work)
update 4/23/24: If you are trying to compile ROOT 6.30, you might need to downgrade to gcc 10.X, see the notes about TBB down below
fftw 3.3.9
fftw 3.3.10 (OSC & Midway; Jason)
gsl 2.7.1 (for ROOT)
gsl 2.8 (Nov. 28 2024; Midway)
ROOT 6.24.00
ROOT 6.26-14 (OSC & Midway)
- OneTBB 2021.12.0 (for ROOT 6.30)
CMake
- Download the source files for CMake.
- Follow the directions on the CMake website.
- Note
- since we're doing a local build, we'll use the
configure script instead of the listed bootstrap script.
- As an example, inside
$HOME/usr/, run wget https://github.com/Kitware/CMake/releases/download/v3.31.1/cmake-3.31.1.tar.gz
tar -xzvf cmake-3.31.1.tar.gz
- Next, compile cmake and then install it,
cd cmake-3.31.1/
mkdir build install
cd build
../configure --prefix=$HOME/usr/cmake-3.31.1/install
- Once configuration is complete, compile and install CMake, You should additionally add this directory to your
$PATH variable: export PATH=$HOME/usr/cmake-3.31.1/install/bin:$PATH
- Note
- On OSC or Midway, you might want to put the above in your
~/.bash_profile and then source it, source ~/.bash_profile.
- Check to make sure that you are using the correct version of CMake:
- Note
- If it's still showing that you are using the default CMake, exit and log in again
gcc 11.1.0
- Download the gcc source. For instance, inside
$HOME/usr, run wget https://github.com/gcc-mirror/gcc/archive/refs/tags/releases/gcc-11.1.0.tar.gz
- Untar the source tarball,
tar -xzvf gcc-11.1.0.tar.gz
- Then install the prerequisites for gcc:
cd gcc-releases-gcc-11.1.0
contrib/download_prerequisites
- Note
- One of the guides I looked at also recommended installing flex separately (on Midway 2 you can do
module load flex; this is not needed on Midway 3.). However, I didn't seem to need to do this on OSC, and I'm not sure how you would go about it without root priviledges, though I imagine it's similar to the process for all the other packages here (download the source and then build by providing an installation prefix somewhere)
- After you have installed the prerequisites, create a build directory:
mkdir build install
cd build
- Then, configure gcc for compilation with
../configure -v --prefix=${HOME}/usr/gcc-releases-gcc-11.1.0/install/ --enable-checking=release --enable-languages=c,c++,fortran --disable-multilib --program-suffix=-11.1
-j8), the compilation on OSC took about 15 min to finish.
- Once gcc is built, we need to set a few environment variables. For instace,
export PATH=$HOME/usr/gcc-releases-gcc-11.1.0/install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/usr/gcc-releases-gcc-11.1.0/install/lib64:$LD_LIBRARY_PATH
- Note
- On OSC or Midway, you might want to put the above in your
~/.bash_profile and then source it, source ~/.bash_profile
- Once this is done, verify that gcc-11.1 is properly installed: Note that plain old might still point to an older version of gcc. This is fine though.
- We also need to make sure cmake uses this compiler:
export CC=$HOME/usr/gcc-releases-gcc-11.1.0/install/bin/gcc-11.1
export CXX=$HOME/usr/gcc-releases-gcc-11.1.0/install/bin/g++-11.1
export FC=$HOME/usr/gcc-releases-gcc-11.1.0/install/bin/gfortran-11.1
FFTW 3.3.9
- Grab the source code for the appropriate version of FFTW
wget https://www.fftw.org/fftw-3.3.10.tar.gz
- Note
- Do not follow the installation instructions on the webpage. Those instructions might work if you have root privileges, but I personally couldn't seem to to get things to work that way.
- Instead, we're going to build fftw with cmake. Untar the fftw source files:
tar -xzvf fftw-3.3.10.tar.gz
- Make a build directory and cd into it:
cd fftw-3.3.10
mkdir build install
cd build
- Now build using cmake, using the flags shown below.
cmake -DCMAKE_INSTALL_PREFIX=$HOME/usr/fftw-3.3.10/install -DBUILD_SHARED_LIBS=ON -DENABLE_OPENMP=ON -DENABLE_THREADS=ON ../
- Next, make and install as usual.
make -j12 && make install
- Now comes the weird part. Remove everything in the build directory, but leave the install directory untouched.
cd $HOME/usr/fftw-3.3.10/
rm -rf build
- Now rebuild fftw, This time, however, we configure the build with an additional flag:
cmake -DCMAKE_INSTALL_PREFIX=$HOME/usr/fftw-3.3.10/install -DBUILD_SHARED_LIBS=ON -DENABLE_OPENMP=ON -DENABLE_THREADS=ON -DENABLE_FLOAT=ON ../
make -j12 && make install
At the end of the day, your fftw install directory should have the following files:
include/fftw3.finclude/fftw3.f03include/fftw3.hinclude/fftw3l.f03include/fftw3q.f03lib64/libfftw3f.solib64/libfftw3f_threads.so.3lib64/libfftw3_omp.so.3.6.9lib64/libfftw3_threads.solib64/libfftw3f_omp.solib64/libfftw3f.so.3lib64/libfftw3f_threads.so.3.6.9lib64/libfftw3.solib64/libfftw3_threads.so.3lib64/libfftw3f_omp.so.3lib64/libfftw3f.so.3.6.9lib64/libfftw3_omp.solib64/libfftw3.so.3lib64/libfftw3_threads.so.3.6.9lib64/libfftw3f_omp.so.3.6.9lib64/libfftw3f_threads.solib64/libfftw3_omp.so.3lib64/libfftw3.so.3.6.9
Why do we have to do things this way? I don't know, I'm bad at computers. Maybe someone more knowledgeable knows. I found that when I didn't do this step, I'd run into errors that pueoBuilder could not find some subset of the required files (either the ones added by building with -DENABLE_FLOAT, or the ones added by building without -DENABLE_FLOAT) (this thread might offer an answer).
- Once fftw has been installed, export your install directory (the one with the
include and lib64 folders) to the following environment variable: export FFTWDIR=$HOME/usr/fftw-3.3.10/install
- Note
- On Midway3, make sure to also puth the following in your
.bash_profile export CPATH=$FFTWDIR/include:$CPATH
- Also, add fftw to the
LD_LIBRARY_PATH. export LD_LIBRARY_PATH=$HOME/usr/fftw-3.3.10/install/lib64:$LD_LIBRARY_PATH
gsl
- gsl is needed for the
mathmore option in ROOT. - Warning
- If you have an outdated version of gsl, ROOT will still compile, but it will skip installing
mathmore and root-config --has-mathmore will return no. To fix this, grab the latest source code for gsl.
wget https://mirror.ibcp.fr/pub/gnu/gsl/gsl-latest.tar.gz
- Untar the files to a directory of your choosing:
tar -xzvf gsl-latest.tar.gz
cd gsl-2.8
- Configure
mkdir build install
cd build
../configure --prefix=$HOME/usr/gsl-2.8/install
make -j12 && make check && make install
- To make sure ROOT can find this installation of gsl, you'll again need to set an environment variable prior to building ROOT,
export GSL_ROOT_DIR=$HOME/usr/gsl-2.8/install/
bin, include, lib and share).
Python 3.9.18
- Download the source code.
wget https://www.python.org/ftp/python/3.9.18/Python-3.9.18.tgz
- Untar as usual,
tar -xzvf Python-3.9.18.tgz
cd Python-3.9.18
Next we will compile Python from source. I used this website for this step.
Make the build and install directories, and then build inside the build directory,
mkdir build install; cd build
and then configure
../configure --prefix=$HOME/usr/Python-3.9.18/install --enable-shared
- Warning
- We must use the flag
--enable-shared.
This is to ensure that shared libraries are built for Python. By not doing this you are preventing any application which wants to use Python as an embedded environment from working, according to this guy. The corresponding error when you try to compile ROOT later on would look like
...can not be used when making a shared object; recompile with -fPIC...
- After configuration, compile as usual
- After everything is installed, you need to add the
lib/ directory inside you python installation to the environment variable $LD_LIBRARY_PATH export LD_LIBRARY_PATH=$HOME/usr/Python-3.9.18/install/lib:$LD_LIBRARY_PATH
- Note
- Without this step I ran into errors when compiling ROOT.
- Also, add the newly install python3 to your
PATH variable. For example, export PATH=$HOME/usr/Python-3.9.18/install/bin:${PATH}
- Next, head over to the binary directory,
cd $HOME/usr/Python-3.9.18/install/bin
pip3 and python3.9.
- If you run you should see the Python interactive terminal, something like:
Python 3.9.18 (main, Apr 5 2024, 22:49:51)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
- We might as well also use
pip3 there to install some useful libraries: ./pip3 install numpy scipy matplotlib
ROOT
Here are the different versions of ROOT.
You might need to additionally install some of the dependencies, but it seems like everything I needed was already installed on my system.
- Download
wget https://root.cern/download/root_v6.26.14.source.tar.gz
tar -xzvf root_v6.26.14.source.tar.gz
- Make the install directory:
cd root-6.26.14
mkdir install
cd build
- Run CMake, but be sure to enable the fortan, mathmore and minuit2 options.
cmake -DCMAKE_INSTALL_PREFIX=$HOME/usr/root-6.26.14/install/ -Dfortran=ON -Dminuit2=ON -Dmathmore=ON ..
- Note
- If you end up with an error related to compiling
XROOTD, then add -Dxrootd=OFF to the original cmake command above. Additionally, on Midway 2 or 3, if it complains about a certain Vdt, same idea, -Dvdt=OFF;
-
if it complains about a missing
libXpm.so and header, turn off X11 with -Dx11=OFF
- Then proceed to start the build:
cmake --build . --target install -j12
- If everything has worked then after the above command finishes running, you should be able to run the following file to finish setting up ROOT:
source $HOME/usr/root-6.26.14/install/bin/thisroot.sh
- Note
- put it in your
.bashrc
- Finally, you should be able to check that ROOT is working; type
root, and you should see (if you turn off X11 you won't be able to do this, but that's okay) ------------------------------------------------------------------
| Welcome to ROOT 6.26/14 https://root.cern |
| (c) 1995-2021, The ROOT Team; conception: R. Brun, F. Rademakers |
| Built for linuxx8664gcc on Nov 28 2023, 00:39:39 |
| From tags/v6-26-14@v6-26-14 |
| With g++-11.1 (GCC) 11.1.0 |
| Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
------------------------------------------------------------------
root [0]
.q to quit the interactive ROOT terminal) and inside a python3 terminal, you should be able to run
pueoBuilder
At this point, the hard work is mostly done, and you should be able to run pueoBuilder.
Perform a prayer to the C++ gods while you're waiting for it to compile, and hopefully at the end of the day you'll have a working set of PUEO software.
Issues I Ran Into
If you already have an existing installation of ROOT, you may still need to recompile to make sure you're using the same c++ standard that the PUEO software is using. I believe the pre-compiled ROOT binaries available through their website are insufficient, though maybe someone else has been able to get those working.
If you're running into errors about c++ standard or compiler version even after you have installed gcc-11.1.0, then for some reason your system isn't recognizing your local installation of gcc-11.1.0. Check the path variables ($PATH and $LD_LIBRARY_PATH) to make sure the gcc-11.1.0 bin/ directory is being searched.
If you're running into an error that looks like:
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
FFTWF_LIB (ADVANCED)
then pueoBuilder can't seem to find your fftw installation (or files that are supposed to be included in that installation), try rebuilding with different flags according to which files it seems to think are missing.
If it seems like pueoBuilder can't seem to find your fftw installation at all (i.e. you're getting some error that looks like missing: FFTW_LIBRARIES or missing: FFTW_INCLUDES), check the environment variables that are supposed to point to your fftw installation ($FFTWDIR) and make sure there are the correct files in the lib/ and include/ subdirectories.
- Note
- Update (6/23/24):
-
The latest version of ROOT (6.30) will fail to compile on OSC unless you manually compile TBB as well. An easy workaround is to simply downgrade to ROOT 6.24, however if you really need ROOT 6.30 you can follow the instructions below to install TBB and compile ROOT.
TBB
- You will first need to downgrade to GCC 10.X. TBB will not compile with GCC 11. This can be done by following the GCC installation isntructions above, except starting with GCC 10 source code instead of GCC 11.
- To install TBB yourself, download the source code (preferably the
.tar.gz file).
- Move the file to the directory where you want to install TBB and untar it with:
tar -xzvf oneTBB-2021.12.0.tar.gz
- Make some build and install directories:
mkdir build install
cd build
- Then configure cmake:
cmake -DCMAKE_INSTALL_PREFIX=/path/to/tbb/install
- Then compile with:
- Once this has finished running, you can add the installation to you
$PATH and $LD_LIBRARY_PATH variables: export PATH=/path/to/tbb/install/bin:$PATH
export LD_LIBRARY_PATH=/path/to/tbb/install/lib64:$LD_LIBRARY_PATH
You can then proceed as normal, except when compiling ROOT you will need one additional cmake flag
(Dbuiltin_tbb=ON):
cmake -DCMAKE_INSTALL_PREFIX=/scratch/wluszczak/root/install/ /scratch/wluszczak/root/root-6.24.00/
-Dfortran=ON -Dminuit2=ON -Dmathmore=ON
-Dbuiltin_tbb=ON
And hopefully this should work. This process is a little bit more involved
than just downgrading ROOT, so try to avoid going down this route unless
absolutely necessary.
1.12.0