Building from source#

Note

Building from source is not necessary for most users or developers. If you are only contributing Python code, a standard installation is sufficient. Building from source is only required if you want to modify or contribute to the C++ core of pyGIMLi (pgcore).

Building on Windows

This is heavily outdated, new version coming soon.

You need a Python installation with the modules numpy and matplotlib. There are minimalistic installers and pre-packaged distributions. We recommend Anaconda <http://www.continuum.io/>_ 64bit but WinPython <http://winpython.github.io/#releases>_ will to the job too.

You also need a Linux-like command shell along with a gcc compiler. Although there might be different solutions (Cygwin, Git Bash, MinGW/MSYS), we only support the MSYS2 (Minimal System 2) hosted at http://www.msys2.org. As computers and modern Windows (>=7) are 64bit we only test this. Avoid installing into strange Windows folders, e.g., c:\ProgramData\mys64 is fine.

After installing MSYS, start the console once so it builds your personal home directory where you find a .bashrc file, e.g. in

… code-block::

c:\ProgramData\mys64\home\YOUR_USERNAME

Edit .bashrc so that the Anaconda or WinPython installation path is added to your default PATH.

e.g.:

… code-block:: bash

export ANACONDA=/c/ProgramData/Anaconda3
export PATH=$PATH:$ANACONDA

This is necessary since gimli needs to know valid python installation and version. Ideally the following one-liner will suffice to compile pyGIMLi in the current directory.

Note: The script will automatically take care of requirements and updates of MSYS2. And also needs to modify/patch some of the llvm system files.

… code:: bash

curl -Ls install.pygimli.org | bash

This script accepts a few more options. For help see

… code:: bash

curl -Ls install.pygimli.org | bash -s help

If everything runs fine, including some tests, the script will tell you some additional PATH and PYTHONPATH settings for your .bashrc to use pygimli inside the console or any IDE like Spyder (coming along with WinPython).

If something goes wrong, please take a look on the error message.

You can also try the following instructions for manual installation.

Manual installation

Make sure to have an updated msys2 environment. Run at least:

… code-block:: bash

pacman -Su
pacman -Sy

to update your local package databases. See https://www.msys2.org/ for further instructions.

To get a complete working toolchain you need some packages installed.

… code-block:: bash

pacman -S make tar git subversion mercurial unzip wget patch

… code-block:: bash

pacman -S mingw-w64-x86_64-cmake mingw-w64-x86_64-gcc mingw-w64-x86_64-gcc-fortran
pacman -S mingw-w64-x86_64-openblas mingw-w64-x86_64-suitesparse
pacman -S mingw-w64-x86_64-doxygen mingw-w64-x86_64-cppunit
pacman -S mingw-w64-x86_64-llvm mingw-w64-x86_64-clang

The rest of the installation is like the linux way with some small differences.

Prepare the directory structure as described above:

The build is performed via cmake. While calling cmake MSYS users should tell using the Unix makefile generator to find the correct gcc versions:

… code-block:: bash

cmake ../gimli -G 'Unix Makefiles'

If cmake complains about missing python stuff, make sure the Python interpreter is in your execution path.

Problems with cmake configuration

If cmake can’t install pygccxml or pyplusplus then you can provide those packages using pip from the anaconda distribution. First make sure the needed scripts are in your path.

… code-block:: bash

export PATH=$PATH:$ANACONDA/Scripts

Then you can install those both packages in your user space

… code-block:: bash

pip install pygccxml –user pip install pyplusplus –user

If cmake complains about misssig numpy, python can’t probably import numpy, which you can test:

… code-block:: bash

python -c 'import numpy'

Probably anaconda additional needs another path setting, don’t ask me why

… code-block:: bash

export PATH=\(PATH:\)ANACONDA/Library/bin

Now python should be able to find numpy and cmake will work as supposed and you can continue the build process.

To build the library, just run

… code-block:: bash

make -j2

You might add the option -jN to use a number of N CPUs in parallel. To build pygimli, run

… code-block:: bash

make pygimli J=2

You might add J=N to use a number of N CPUs in parallel. Building pygimli takes some time and you can grab a coffee (or two). If it finishes without errors you just need to set the environment: (note that pygimli is still built in-place, in pygimli/core)

… code-block:: bash

export PYTHONPATH=$PYTHONPATH:$HOME/src/gimli/gimli/python
export PATH=$PATH:$HOME/src/gimli/build/lib
export PATH=$PATH:$HOME/src/gimli/build/bin

Compiled binaries will be written to build/bin.

You can test the pygimli build with:

… code-block:: bash

python -c 'import pygimli as pg; print(pg.version())'

Building on MAC OSX

The current working solution is based on this discussion on GitHub. Many thanks to Robin Thibaut!

# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Install dependencies via brew
brew install cmake
brew install wget
brew install mercurial

# Install Miniforge
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

conda create -n pygimli_env python
conda activate pygimli_env
conda install -c conda-forge boost numpy scipy matplotlib openblas suitesparse

# Clone the pygimli repository
git clone https://github.com/gimli-org/gimli.git source
mkdir build
cd build

PYTHON_EXEC=$(which python3)
PYTHON_INC=$(python3 -c 'import sysconfig; print(sysconfig.get_path("include"))')
PYTHON_LIB=$(python3-config --configdir)

export CPLUS_INCLUDE_PATH=$PYTHON_INC

cmake -DPYTHON_EXECUTABLE=$PYTHON_EXEC -DPYTHON_LIBRARY=$PYTHON_LIB -DPYTHON_INCLUDE_DIR=$PYTHON_INC ../source
make -j 8
make pygimli J=8

Troubleshooting

If you encounter problems, you may have to specify some paths manually, e.g.:

cmake -DPYTHON_EXECUTABLE=$PYTHON_EXEC -DPYTHON_LIBRARY=$PYTHON_LIB
    -DPYTHON_INCLUDE_DIR=$PYTHON_INC \
    -DUMFPACK_LIBRARIES=~/minforge3/base/lib/libumfpack.dylib \
    -DUMFPACK_INCLUDES=~/minforge3/base/include \
    -DCHOLMOD_LIBRARIES=~/minforge3/base/lib/libcholmod.dylib \
    -DCHOLMOD_INCLUDE_DIRS=~/minforge3/base/include \
    -DBLAS_openblas_LIBRARY=~/minforge3/base/lib/libopenblas.dylib \
    -DOpenBLAS_INCLUDE_DIR=~/minforge3/base/include \
    -DBoost_PYTHON_LIBRARY=~/minforge3/base/lib/libboost_python310.dylib \
    ../source

Building on Linux

Using internal workflow

We provide a workflow script for our CI pipeline/action runner which may be use also local for all platforms.

Prepare workspace:

mkdir -p gimli
git clone https://github.com/gimli-org/gimli.git
# optionally change to the development branch
git checkout dev

Running workflow:

bash gimli/.workflow.sh install

The workflow have several targets which you might try:

bash gimli/.workflow.sh help

If something goes wrong you can try the manual compilation.

Manual compiling with a virtual environment

If you don’t want to use the conda environment we encourage the use of a virtual environment. Assuming you have a proper build toolchain and the required libraries (see Installation on Ubuntu below) installed.

First we need to create a root directory for our installation, e.g., $HOME/src/gimli and get the source code:

mkdir -p gimli
cd gimli
git clone https://github.com/gimli-org/gimli.git
# optionally change to the development branch
git checkout dev
# create a virtual environment for pyGIMLi, this can be at any place with any name
# if you want easy VScode support consider gimli/.venv
deactivate # in case there is another venv active
python -m venv venv-build --prompt=gimli-build
# activate the venv
source venv-build/bin/activate
# update pip is always a good idea
python -m pip install -U pip
# install gimli as editable with its dependencies into to venv
pip install -e ./gimli/[build]

We need to compile the C++ part of pyGIMLi, which is currently done with cmake and not with the pip internal build system. We recommend an out of source build:

mkdir -p build
cd build
cmake ../gimli
make -j 4 gimli
make pygimli J=4
cd ..

There is no more need to change the PATH or LD_LIBRARY_PATH. In fact, check to remove any prior changes to these environment variables from older installations. If the build was successful he copy the compiled libraries back into the source tree which is already editable known to the venv, so you can test the pygimli build with:

python -c 'import pygimli as pg; pg.version()'
python -c 'import pygimli as pg; print(pg.Report())'

Note, if you test like this, ensure there is no actual path with a name pygimli in your current directory or he will test this path as package instead. This is a sideeffect of the --editable install and such the test will probably fail.

As long nothing changes in the C++ part of pygimli you can just update pyGIMLi but just pulling the latest changes from git. If you end the terminal session you can reactivate the venv with:

source venv-build/bin/activate

Example Installation on Ubuntu

Last try on Ubuntu 22.04.03 (23-11-14)

sudo apt-get install build-essential g++ subversion git cmake \
            python3-dev python3-matplotlib python3-numpy python3-pyqt5 \
            python3-scipy libboost-all-dev libedit-dev \
            libsuitesparse-dev libopenblas-openmp-dev libumfpack5 \
            libomp-dev doxygen \
            libcppunit-dev clang

Rest see above.

Useful cmake settings

You can rebuild and update all local generated third party software by setting the CLEAN environment variable:

CLEAN=1 cmake ../gimli

Use an alternative C++ compiler:

CC=clang CXX=clang++ cmake ../gimli

Build the library with debug and profiling flags:

cmake ../gimli -DCMAKE_BUILD_TYPE=Debug

Build the library with GCC address sanitizer check:

cmake ../gimli -DCMAKE_BUILD_TYPE=Debug -DASAN=1

Useful make commands

Show more verbose build output:

make VERBOSE=1

Dependencies

To build pyGIMLi from source, the following tools are required:

  • subversion, git, mercurial, wget, tar

  • cmake >= 2.8.10

  • gcc >= 4.4

  • >=Python-3.5 | >=Python-2.7

  • numpy-dev

  • >=matplotlib-3.0

  • >=clang+±3.6.0 (3.7.0, 3.8.0)

  • libclang-3.7-dev

  • >=llvm-3.6.0 (3.7.0, 3.8.0)

  • libz-dev

  • python-setuptools

    tested on:

    • gentoo x86_64: gcc-4.4.5, gcc-4.5.3, gcc-4.5.4, gcc-4.9.2 gcc-5.3.0

    • Debian 3.2.46-1 x86_64: gcc-4.7.2

    • Ubuntu 16.04 LTS with gcc-5.4.0

    • Arch Linux gcc-5.2.0

    • CentOS

    • MinGW32: gcc-4.6.2/4, gcc-4.7.2, gcc-5.2.0

    • MinGW64: gcc-4.5.4, gcc-5.2.0, gcc-6.3.0, gcc-7.1.0

Optional Prerequisites

These tools can be installed system-wide with your native package manager (i.e. apt-get). If not found, the provided build scripts will automatically download and compile the necessary components.

Prerequisites automatically installed by the installer

These tools are required to create the Python bindings and are likely to be outdated in your distribution and should therefore not be installed system-wide. The build scripts will install them automatically.

  • castxml

  • pygccxml

  • pyplusplus