[molpro-user] 2012.1 build with slurm/mvapich2

[Andy May]

Martin,

I can't comment on the problem you see since we don't have such a setup, 
but I have added environment variable MV2_ENABLE_AFFINITY to the envlist 
in bin/molpro, this change will appear in the nightly builds in the next 
day or so.

I'm not certain why you needed to add OMP_NUM_THREADS to the list since 
the main executable (bin/molpro.exe) sets the number of threads based on 
the command line -t option. Run:

grep -i molpro_omp_threads src/*/*.F*

to see exactly what happens.

The OMP_NUM_THREADS variable should be detected by bin/molpro and then 
passed as a command line option '-t' to bin/molpro.exe. If one passes -t 
to bin/molpro then the script should pass instead pass it on as -t to 
bin/molpro.exe. Try running with -v to confirm that this option gets 
passed on via the command line.

Best wishes,

Andy

On 02/12/12 16:51, Martin Diefenbach wrote:
> Dear all,
>
> we would like to use molpro in conjunction with the SLURM batch queueing
> system, that is, in particular we are trying to build Molpro-2012.1 on
> an AMD Opteron QDR-Infiniband Linux-cluster (managed by the SLURM batch
> queueing system) using the following configuration:
>
>   INTEL ifort/icc 12.1
>   MKL 10.3
>   GLOBAL ARRAYS 5-1-1
>   MVAPICH2-1.8
>   SLURM (2.4.1)
>
> To exploit both, the fast Infiniband interconnect as well as the
> intra-node shared-memory multiple CPU architecture, we were aiming at an
> OpenMP/MPI Hybrid build.
>
> Compilation of the code works fine, and MPI/OpenMP-hybrid molpro jobs
> start to run correctly; e.g. "molpro -n 4 -t 2 test.inp" runs on 4 CPUs,
> each spawning 2 threads = total of 8 processes, and produces some
> regular output. They do, however, cause the nodes to crash just before
> the variable memory is released (by starting a second instance of
> slurmstepd, i.e., more precisely a second set of molpro.exe's, which
> consume all of the resident memory and lead to a blow-up).
>
> Can anyone confirm this or has anyone succeeded in a similar
> environment? I attach the current CONFIG with this email.
>
> Cheers,
> Martin
>
> PS: After several many trials, the latest configuration we used is the
> following (utilising the nightly build of molpro):
>
> MVAPICH2-1.8:
> =============
> ./configure \
>    --prefix=${PREFIX}/mvapich2 \
>    --with-device=ch3:mrail \
>    --with-rdma=gen2 \
>    --enable-shared \
>    --disable-xrc \
>    --with-pm=no \
>    --with-pmi=slurm \
>    --with-limic2=/opt/limic2-0.5.5 \
>    --with-hwloc=/cm/shared/apps/hwloc/1.4.2
>
> GA-5-1-1:
> =========
> LIBMPI="-I${SLURM_HOME}/include -I${MPI_HOME}/include
> -I/opt/limic2-0.5.5/include \
>   -L${SLURM_HOME}/lib -L${MPI_HOME}/lib -L/opt/limic2-0.5.5/lib \
>   -lmpichf90 -lmpich -lopa -lmpl -lpmi -lpthread -llimic2 -lrdmacm
> -libverbs -libumad -ldl -lrt -lnuma"
> BLAS_LIBS="-L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm"
> ./configure \
>      MPIEXEC="srun -n %NP%" \
>      CC=icc F77=ifort CXX=icpc \
>      CFLAGS="-DLARGE_FILES -DMKL_ILP64 -openmp" \
>      FFLAGS="-openmp" \
>      --with-blas8="${BLAS_LIBS}" \
>      --with-mpi="${LIBMPI}" \
>      --with-tcgmsg \
>      --with-openib \
>      --prefix=${PREFIX}/ga
>
> MOLPRO-2012.1:
> ==============
> ./configure \
>    -batch \
>    -prefix "${PREFIX}" \
>    -blaspath "${MKLROOT}/lib/intel64" \
>    -openmp \
>    -mpp \
>    -mppbase "../ga-5-1-1" \
>    -ifort -icc
>
> The CONFIG file is then edited/adapted to SLURM and thread-safe MKL by
> changing the two corresponding variables into
>   LAUNCHER="srun -n %n %x"
>   BLASLIB="-L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm"
>
> Also, molpro.sh is edited to include "OMP_NUM_THREADS
> MV2_ENABLE_AFFINITY" with $envlist.
>
>
>
> _______________________________________________
> Molpro-user mailing list
> Molpro-user at molpro.net
> http://www.molpro.net/mailman/listinfo/molpro-user
>