[molpro-user] Problems with a large eom-ccsd calculation

[Manhui Wang]

Dear Evgeniy,


It appears your CONFIG is not right. As I mentioned in previous email:

./configure -batch  -icc -ifort -blaspath
/software/intel/mkl/10.2.5.035/lib/em64t -mpp -mppbase ${GA_path} -var
LIBS="-libverbs"

Please note ${GA_path} is not the GA installation directory, but the
original GA directory where you type"configure...make". Molpro configure
takes some info from ${GA_path} to determine the GA mode and the
underlying MPI library etc.
-var LIBS="-libverbs" will enable IntelMPI to be linked with Infiniband
library if your IntelMPI on the machine can't do this automatically.
Once the CONFIG is generated, we don't recommend to modify it manually
since it may introduce unexpected problems.

Best wishes,
Manhui

Evgeniy Gromov wrote:
> Dear Manhui,
> 
> Thanks for your response.
> There is unfortunately no progress with GA. I compiled molpro
> but the binary doesn't work at all, although the compilation
> went OK. There is somehow again no ABDATA in lib, but it doesn't
> complain to that. What it complains looks strange:
> 
> Attempting to use an MPI routine before initializing MPICH
> 
> 
> Well I used intel MPI and I don't understand why it refers to MPICH.
> 
> Just in case my CONFIG looks the following:
> 
> # MOLPRO CONFIG generated at Di 5. Apr 10:21:18 CEST 2011, for host frbw3
> 
> CONFIGURE_OPTIONS="-i8" "-openmp" "-icc" "-ifort" "-nohdf5" "-nocuda"
> "-mpp" "-mppbase" "/bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install"
> "-blaspath" "/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t"
> 
> AR=ar
> ARCHNAME=Linux/x86_64
> ARFLAGS=-rS
> AWK=awk
> BIBTEX=/usr/bin/bibtex
> BLASLIB=-L/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t
> -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core
> BUILD=p
> CAT=cat
> CC=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/bin/intel64/icc
> CCVERSION=11.1
> CC_FRONT=
> CDEBUG=-g $(addprefix $(CDEFINE),_DEBUG)
> CDEFINE=-D
> CFLAGS=-ftz -vec-report0 -DINT64 -DZLIB -openmp
> CLEARSPEEDLIB=
> CMPPINCLUDE=/bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install/include
> COPT=-O2
> COPT0=-O0
> COPT1=-O1
> COPT2=-O2
> COPT3=-O3
> CP=cp -p
> CPROFILE=-p
> CSCN=cscn
> CSFLAGS=-O3 -I. --dynamic
> CUDACC=
> CUDACCVERSION=
> CUDACDEBUG=-g $(addprefix $(CUDACDEFINE),_DEBUG)
> CUDACDEFINE=-D
> CUDACFLAGS=
> CUDACOPT=
> CUDACOPT0=-O0
> CUDACOPT1=-O1
> CUDACOPT2=-O2
> CUDACOPT3=-O3
> CUDACPROFILE=-p
> CXX=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/bin/intel64/icpc
> CXXFLAGS=$(CFLAGS)
> DOXYGEN=/usr/bin/doxygen
> ECHO=echo
> EXPORT=export
> F90FLAGS=
> FC=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/bin/intel64/ifort
> FCVERSION=11.1
> FDEBUG=-g $(addprefix $(FDEFINE),_DEBUG)
> FDEFINE=-D
> FFLAGS=-i8 -r8 -pc64 -auto -Vaxlib -vec-report0 -cxxlib -openmp
> FOPT=-O3
> FOPT0=-O0
> FOPT1=-O1
> FOPT2=-O2
> FOPT3=-O3
> FPP=-fpp
> FPROFILE=-p
> FSTATIC=
> FTCFLAGS=molpro unix unix-i8 Linux lapack mpp blas
> HDF5INCLDE=
> HDF5LIB=
> HOSTFILE_FORMAT=
> INSTBIN=/bwfs/scratch/temphomes/hd/f81/molpro2010.1-install/bin
> INSTHTML=/bwfs/scratch/temphomes/hd/f81/molpro2010.1-install/html
> INSTLIB=/bwfs/scratch/temphomes/hd/f81/molpro2010.1-install/lib/molprop_2010_1_Linux_x86_64_i8
> 
> INST_PL=0
> INTEGER=8
> LAPACKLIB=
> LATEX2HTML=
> LAUNCHER=/opt/bwgrid/mpi/mpiexec/0.84/bin/mpiexec -n %n %x
> LD_ENV=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/lib/intel64:/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t
> 
> LD_ENVNAME=LD_LIBRARY_PATH
> LIBRARY_SUFFIX=a
> LIBS=-lz
> LIBS_FRONT=
> LINKOPT=
> LINKOPT_FRONT=
> LN=ln -s
> MACROS=MOLPRO MOLPRO_ifort MOLPRO_f2003 GA_TOOLS GA_ GA_VERSION_GE_5
> _I8_ BLAS_INT=8 LAPACK_INT=8 MOLPRO_FORCE_VECTOR MOLPRO_NEXTSCALAR
> MOLPRO_NO_RECURRENCE MOLPRO_NOVECTOR MOLPRO_SHORTLOOP _MOLCAS_MPP_
> MAKEDEPEND_OPTIONS=
> MAKEINDEX=/usr/bin/makeindex
> MAPLE=
> MKDIR=mkdir -p
> MODULE_FLAG=-I
> MODULE_SUFFIX=mod
> MPILIB=-L/opt/bwgrid/compiler/intel/ct_4.0/impi/4.0.0.028/intel64/lib
> -lmpi -lmpigf -lmpigi -lpthread -L/usr/lib64 -libverbs -libcommon
> -libumad -lpthread
> MPPLIB=-L/bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install/lib/LINUX64
> -lga -larmci
> OBJECT_SUFFIX=o
> OPT0=copyc6.f
> OPT1=nevpt2_optrpc.f explicit_util.f artwo2.f drv2el_l3ext_lmp2g.f
> drv2el_l3ext_cen.f rmp2_f12_drv2.f90 ri_lmp2g.f df_llmp2.f readdump.f
> OPT2=integrals.f90 RL4gen1.f basis_integral_shells.f
> OPT3=
> PAPER=a4paper
> PARSE=parse-Linux-x86_64-i8.o
> PDFLATEX=/usr/bin/pdflatex -halt-on-error
> PNAME=molprop_2010_1_Linux_x86_64_i8
> PTSIZE=11
> RANLIB=ranlib
> RM=rm -rf
> SHELL=/bin/sh
> STRIP=strip
> SUFFIXES=f F f90 F90 c cpp
> TAR=tar -cf
> UNTAR=tar -xf
> VERBOSE=@
> VERSION=2010.1XSD=/usr/bin/xmllint --noout --schema
> XSLT=/usr/bin/xsltproc
> YACC=bison -b y
> 
> .SUFFIXES:
> MAKEFLAGS+=-r
> ifneq ($(LD_ENVNAME),)
> $(LD_ENVNAME):=$(LD_ENV):$($(LD_ENVNAME))
> endif
> 
> 
> 
> 
> 
> Best regards,
> Evgeniy
> 
> Manhui Wang wrote:
>> Dear Evgeniy,
>>
>> I have not checked your output carefully, which shows 800MWord (6.4GB
>> per process):
>>
>> Variable memory set to  800000000 words,  buffer space   230000 words
>>
>> I run the job  with 32 processes on 4 nodes, and it might crash
>> eventually due to lack of memory you mentioned.
>>
>> Could you please update your progress with GA? On my side, I will look
>> into my code related to helper. It might exist a bug which is exposed
>> when very large global data structure is used.
>>
>> Best wishes,
>> Manhui
>>
>>
>>
>> Evgeniy Gromov wrote:
>>> Dear Manhui,
>>>
>>> Many thanks for your help with the compilation problems and
>>> with the problem in the large eom-ccsd calculation.
>>> Sorry I forgot to tell you about the memory. I usually specify
>>> the memory in the command line using -m option. For that large
>>> eom-ccsd job I used 800m (-m800m) because 200m was not enough for
>>> the CCSD step. As I wrote the job passed the CCSD and crashed
>>> in the beginning of the EOM-CCSD. Actually, are you running it
>>> on 1 CPU? I haven't tried it to run on 1 CPU as it will likely
>>> take ages cause many MOs. Well I am going to rerun it using
>>> a new binary compiled with GA.
>>>
>>> Best wishes,
>>> Evgeniy
>>>
>>>
>>> Manhui Wang wrote:
>>>> Dear Evgeniy,
>>>>
>>>> I can't see the error you mentioned with MPI-2 Molpro2010.1 PL20 so
>>>> far.
>>>> I tried your input without any modification first, and it failed
>>>> immediately because you have not set memory in input (the default is
>>>> only 8MWord). I have added
>>>>
>>>> memory,200,m
>>>>
>>>> in input, it appears to work fine so far (still running).  Could you
>>>> please try the new input on your side?
>>>>
>>>> Best wishes,
>>>> Manhui
>>>>
>>>>
>>>> Evgeniy Gromov wrote:
>>>>> Manhui,
>>>>>
>>>>> I attached wrong files in the previous email. Here are the correct
>>>>> input and output files. In addition I removed all the MO vectors in
>>>>> the output to make the size smaller.
>>>>>
>>>>> Best regards,
>>>>> Evgeniy
>>>>>
>>>>> Manhui Wang wrote:
>>>>>> Dear Evgeniy,
>>>>>>
>>>>>> Could you please provide the inputs/outputs? I will see whether I can
>>>>>> reproduce the problem firstly.
>>>>>>
>>>>>> Thanks,
>>>>>> Manhui
>>>>>>
>>>>>> Evgeniy Gromov wrote:
>>>>>>> Dear Molpro Community,
>>>>>>>
>>>>>>> I faced problems when running a large eom-ccsd calculation (466 MOs,
>>>>>>> no symmetry) with molpro2010.1 in parallel (MPI-2). The calculation
>>>>>>> crashed after the CCSD (ground state) step had been done. No error
>>>>>>> message was present in the output. In the err file there is the
>>>>>>> following diagnostic (though unclear to me):
>>>>>>>
>>>>>>> ERROR in twosided_helpga_locate_server: over range!
>>>>>>> ilo=1,ihigh=148114575
>>>>>>>
>>>>>>> Any idea what can be the problem.
>>>>>>>
>>>>>>> Well, it seems to relate to helper process. I suspect that in the
>>>>>>> case
>>>>>>> of GA it would work, although there might be some other problems.
>>>>>>> Just in case the same run with 296 MO (another basis set) ran fine.
>>>>>>>
>>>>>>> Also, I would like to thank Manhui for the response to my previous
>>>>>>> post:
>>>>>>> "Is there always a helper process in parallel calculations with
>>>>>>> molpro2010.1 ?"
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Evgeniy
>>>>>>>
>>>>>>>
>>>
>>
> 
> 

-- 
-----------
Manhui  Wang
School of Chemistry, Cardiff University,
Main Building, Park Place,
Cardiff CF10 3AT, UK