[molpro-user] Fwd: Problems with parallel build on AMD Opteron(tm) Processor 6376

Thu Jan 29 11:12:13 GMT 2015

Hi,

This looks like you are hit by Open MPI's automatic core binding which started in 1.7.x.

Either disable Open MPI's automatic core binding or provide a disjunct list of cores for each job:

$ mpiexec --bind-to none ...

This will no change any binding which was already setup by your queuing system though. Although the behavior is different in some way: the queuingsystem will most often bind a set of processes to a set of cores - i.e. the scheduler of the OS could shift the processes around inside this set. Open MPI on the other hand will bind each process to a unique core.

Also note, that starting in Open MPI 1.8.2 they do a more intensive network scan for possible routes between the machines which might (depending on your network setup) lead to some delay in the range of 2 minutes after `mpiexec` was issued before the application finally starts.

Best would be to have a closer look to these issues outside of Molpro first.

-- Reuti

PS: Using Open MPI 1.6.5 might also be worth to be tested, as there these issues aren't present.

> Am 29.01.2015 um 11:26 schrieb molpro-user <molpro-user at molpro.net>:
> 
> 
> 
> == == == == == == Forwarded message == == == == == == 
> From : Robert Polly<polly at kit.edu>
> To : <molpro-user at molpro.net>
> Date : Fri, 23 Jan 2015 12:50:51 +0000
> Subject : Problems with parallel build on AMD Opteron(tm) Processor 6376
> == == == == == == Forwarded message == == == == == == 
> 
> 
> Dear MOLPRO community, 
> we installed MOLPRO on our new Opteron cluster (64 CPUs per node) 
> 
> CPU: AMD Opteron(tm) Processor 6376 
> Compiler: ifort/icc 
> Openmpi: openmpi-1.8.3 
> Molpro: 2012.1.18 
> 
> CONFIG file: 
> 
> # MOLPRO CONFIG generated at Fri Jan 9 10:31:38 MET 2015, for host 
> master.hpc1.ine, SHA1=50d6e5f7071a51146f1443020887856fd3d38933 
> 
> CONFIGURE_OPTIONS="-icc" "-ifort" "-mpp" "-openmpi" "-mppbase" 
> "/home/polly/molpro/TEST/Molpro.2012.1.18.par/src/openmpi-install/include" 
> 
> AR=ar 
> ARCHNAME=Linux/x86_64 
> ARFLAGS=-rS 
> AWK=awk 
> BIBTEX= 
> BLASLIB=-L/pub/hpc/module/compilers/intel/xe2015/composer_xe_2015.0.090/mkl/lib/intel64 
> -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core 
> BUILD=p 
> CAT=cat 
> CC=/pub/hpc/module/compilers/intel/xe2015/composer_xe_2015.0.090/bin/intel64/icc 
> CCVERSION=15.0.0 
> CC_FRONT= 
> CDEBUG=-g $(addprefix $(CDEFINE),_DEBUG) 
> CDEFINE=-D 
> CFLAGS=-ftz 
> -I/home/polly/molpro/TEST/Molpro.2012.1.18.par/src/openmpi-install/include 
> CLDFLAGS= 
> CLEAN=echo 'target clean only available with git cloned versions, please 
> unpack the tarball again' 
> CMPPINCLUDE=/home/polly/molpro/TEST/Molpro.2012.1.18.par/src/openmpi-install/include 
> COPT=-O2 
> COPT0=-O0 
> COPT1=-O1 
> COPT2=-O2 
> COPT3=-O3 
> CP=cp -p 
> CPROFILE=-p 
> CUDACC= 
> CUDACCVERSION= 
> CUDACDEBUG=-g $(addprefix $(CUDACDEFINE),_DEBUG) 
> CUDACDEFINE=-D 
> CUDACFLAGS= 
> CUDACOPT= 
> CUDACOPT0=-O0 
> CUDACOPT1=-O1 
> CUDACOPT2=-O2 
> CUDACOPT3=-O3 
> CUDACPROFILE=-p 
> CXX=/pub/hpc/module/compilers/intel/xe2015/composer_xe_2015.0.090/bin/intel64/icpc 
> CXXFLAGS=$(CFLAGS) 
> DOXYGEN=/bin/doxygen 
> ECHO=/bin/echo 
> EXPORT=export 
> F90FLAGS=-stand f03 
> FC=/pub/hpc/module/compilers/intel/xe2015/composer_xe_2015.0.090/bin/intel64/ifort 
> FCVERSION=15.0.0 
> FDEBUG=-g $(addprefix $(FDEFINE),_DEBUG) 
> FDEFINE=-D 
> FFLAGS=-i8 -pc64 -auto -warn nousage -align array32byte -cxxlib 
> FLDFLAGS= 
> FOPT=-O3 
> FOPT0=-O0 
> FOPT1=-O1 
> FOPT2=-O2 
> FOPT3=-O3 
> FPROFILE=-p 
> FSTATIC= 
> HOSTFILE_FORMAT=%N 
> INSTALL_FILES_EXTRA=src/openmpi-install/bin/mpirun 
> src/openmpi-install/bin/orterun 
> INSTBIN= 
> INST_PL=0 
> INTEGER=8 
> LAPACKLIB= 
> LATEX2HTML= 
> LAUNCHER=/home/polly/molpro/TEST/Molpro.2012.1.18.par/src/openmpi-install/bin/mpirun 
> --mca mpi_warn_on_fork 0 -machinefile %h -np %n %x 
> LD_ENV=/pub/hpc/module/compilers/intel/xe2015/composer_xe_2015.0.090/compiler/lib/intel64:/pub/hpc/module/compilers/intel/xe2015/composer_xe_2015.0.090/mkl/lib/intel64 
> LD_ENVNAME=LD_LIBRARY_PATH 
> LIBRARY_SUFFIX=a 
> LIBS=-lpthread 
> /home/polly/molpro/TEST/Molpro.2012.1.18.par/src/boost-install/lib/libboost_system.a 
> /home/polly/molpro/TEST/Molpro.2012.1.18.par/src/boost-install/lib/libboost_thread.a 
> -lrt 
> LN=ln -s 
> MACROS=MOLPRO NDEBUG MOLPRO_f2003 MOLPRO_bug3990 MPI2 HAVE_BOOST_THREADS 
> HAVE_SSE2 _I8_ MOLPRO_INT=8 BLAS_INT=8 LAPACK_INT=8 MOLPRO_AIMS 
> MOLPRO_NECI _MOLCAS_MPP_ MOLPRO_BLAS MOLPRO_LAPACK 
> MAKEDEPEND_OPTIONS= 
> MAKEINDEX= 
> MAPLE= 
> MAX_INCREMENT_LIBRARY=0 
> MKDIR=mkdir -p 
> MODULE_FLAG=-I 
> MODULE_SUFFIX=mod 
> MPILIB=-I/home/polly/molpro/TEST/Molpro.2012.1.18.par/src/openmpi-install/lib 
> -Wl,-rpath 
> -Wl,/home/polly/molpro/TEST/Molpro.2012.1.18.par/src/openmpi-install/lib 
> -Wl,--enable-new-dtags -L/home/polly/molpro/TEST\ 
> /Molpro.2012.1.18.par/src/openmpi-install/lib -lmpi_usempif08 
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm 
> -lpciaccess -ldl -lrt -losmcomp -libverbs -lrdmacm -lutil -lpsm_infinipath 
> MPPLIB= 
> OBJECT_SUFFIX=o 
> OPT0=B88.F copyc6.F 
> OPT1=explicit_util.F avcc.F koopro4.F dlaed4.F frequencies.F optg.F 
> OPT2=tstfnc.F dftgrid.F mrf12_singles.F90 basis_integral_shells.F 
> integrals.F90 geminal.F surface.F gcc.F90 
> OPT3= 
> PAPER=a4paper 
> PARSE=parse-Linux-x86_64-i8.o 
> PDFLATEX= 
> PNAME=molprop_2012_1_Linux_x86_64_i8 
> PREFIX=/usr/local/molpro/molprop_2012_1_Linux_x86_64_i8 
> PTSIZE=11 
> PYTHON=/bin/python 
> RANLIB=ranlib 
> RM=rm -rf 
> SHELL=/bin/sh 
> STRIP=strip 
> SUFFIXES=F F90 c cpp 
> TAR=tar -cf 
> UNTAR=tar -xf 
> VERBOSE=@ 
> VERSION=2012.1 
> XSD=/bin/xmllint --noout --schema 
> XSLT=/bin/xsltproc 
> YACC=bison -b y 
> 
> .SUFFIXES: 
> MAKEFLAGS+=-r 
> ifneq ($(LD_ENVNAME),) 
> $(LD_ENVNAME):=$(LD_ENV):$($(LD_ENVNAME)) 
> endif 
> 
> 
> We encounter the problem, that when 2 MOLPRO jobs run on one node 
> with 8 processors each, they run on the same 8 processors already 
> allocated by 
> thge first job, although there are still 56 free processors on the machine. 
> 
> Any suggestions how to solve the problem? 
> 
> Best regards, 
> Robert 
> 
> -- 
> 
> ********************************************************************* 
> 
> Karlsruher Institut für Technologie (KIT) 
> Institut fuer Nukleare Entsorgung 
> 
> Dr. Robert Polly 
> 
> Quantenchemie 
> 
> Institut fuer Nukleare Entsorgung (INE), Campus Nord, Gebaeude 712, 
> Postfach 3640, 76021 Karlsruhe, Germany 
> 
> 0049-(0)721-608-24396 
> 
> email: polly at kit.edu 
> www: http://www.fzk.de/ine 
> 
> KIT - Universität des Landes Baden-Württemberg und 
> nationales Großforschungszentrum in der Helmholtz-Gemeinschaft 
> 
> ********************************************************************* 
> 
> 
> 
> 
> <CONFIG>_______________________________________________
> Molpro-user mailing list
> Molpro-user at molpro.net
> http://www.molpro.net/mailman/listinfo/molpro-user