[molpro-user] Problems with a large eom-ccsd calculation

Tue Apr 5 14:58:02 BST 2011

Dear Manhui,

GA runs OK (I did test.x for 16 CPUs).
To run molpro configure in the fresh molpro2010.1
I use your example (with the corrected paths):

./configure -batch -icc -ifort -mpp -mppbase 
/bwfs/scratch/temphomes/hd/f81/prog/ga-5-0-2 -var LIBS="-libverbs" 
-blaspath /opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t

molpro configure crashes then after the following:

ga_GA_MP_LIBS     = -lmpi -lmpigf -lmpigi -lpthread
ga_GA_MP_LDFLAGS  = 
-L/opt/bwgrid/compiler/intel/ct_4.0/impi/4.0.0.028/intel64/lib
ga_GA_MP_CPPFLAGS = 
-I/opt/bwgrid/compiler/intel/ct_4.0/impi/4.0.0.028/intel64/include
ga_TARGET         = LINUX64
ga_MSG_COMMS      = TCGMSGMPI
ga_prefix         = /bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install

MPPLIB=
parallel build specified but unable to find Global Arrays

Result so far of configure written to CONFIG.errout

Apperently molpro configure finds the config.log in the ga-5-0-2
directory. But what it needs then I don't understand. What is
this MPPLIB ?

Best,
Evgeniy

Manhui Wang wrote:
> Dear Evgeniy,
> 
> Could you please check your GA is working properly?
> Could you please provide more details about how do you run configure in
> a fresh Molpro2010.1 PL20?
> 
> Best wishes,
> Manhui
> 
> Evgeniy Gromov wrote:
>> Dear Manhui,
>>
>> If I specify the original GA directory configure cannot find some
>> libraries whatever and crashes:
>>
>> ga_GA_MP_LIBS     = -lmpi -lmpigf -lmpigi -lpthread
>> ga_GA_MP_LDFLAGS  =
>> -L/opt/bwgrid/compiler/intel/ct_4.0/impi/4.0.0.028/intel64/lib
>> ga_GA_MP_CPPFLAGS =
>> -I/opt/bwgrid/compiler/intel/ct_4.0/impi/4.0.0.028/intel64/include
>> ga_TARGET         = LINUX64
>> ga_MSG_COMMS      = TCGMSGMPI
>> ga_prefix         = /bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install
>>
>> MPPLIB=
>> parallel build specified but unable to find Global Arrays
>>
>>
>> It looks like the structure of the GA directory has been changed
>> in the latest release. At least there is neither include no lib
>> directories in the GA directory. Should I take some older GA realese
>> then? Which one?
>>
>> Best,
>> Evgeniy
>>
>> Manhui Wang wrote:
>>> Dear Evgeniy,
>>>
>>>
>>> It appears your CONFIG is not right. As I mentioned in previous email:
>>>
>>> ./configure -batch  -icc -ifort -blaspath
>>> /software/intel/mkl/10.2.5.035/lib/em64t -mpp -mppbase ${GA_path} -var
>>> LIBS="-libverbs"
>>>
>>> Please note ${GA_path} is not the GA installation directory, but the
>>> original GA directory where you type"configure...make". Molpro configure
>>> takes some info from ${GA_path} to determine the GA mode and the
>>> underlying MPI library etc.
>>> -var LIBS="-libverbs" will enable IntelMPI to be linked with Infiniband
>>> library if your IntelMPI on the machine can't do this automatically.
>>> Once the CONFIG is generated, we don't recommend to modify it manually
>>> since it may introduce unexpected problems.
>>>
>>> Best wishes,
>>> Manhui
>>>
>>> Evgeniy Gromov wrote:
>>>> Dear Manhui,
>>>>
>>>> Thanks for your response.
>>>> There is unfortunately no progress with GA. I compiled molpro
>>>> but the binary doesn't work at all, although the compilation
>>>> went OK. There is somehow again no ABDATA in lib, but it doesn't
>>>> complain to that. What it complains looks strange:
>>>>
>>>> Attempting to use an MPI routine before initializing MPICH
>>>>
>>>>
>>>> Well I used intel MPI and I don't understand why it refers to MPICH.
>>>>
>>>> Just in case my CONFIG looks the following:
>>>>
>>>> # MOLPRO CONFIG generated at Di 5. Apr 10:21:18 CEST 2011, for host
>>>> frbw3
>>>>
>>>> CONFIGURE_OPTIONS="-i8" "-openmp" "-icc" "-ifort" "-nohdf5" "-nocuda"
>>>> "-mpp" "-mppbase" "/bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install"
>>>> "-blaspath" "/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t"
>>>>
>>>> AR=ar
>>>> ARCHNAME=Linux/x86_64
>>>> ARFLAGS=-rS
>>>> AWK=awk
>>>> BIBTEX=/usr/bin/bibtex
>>>> BLASLIB=-L/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t
>>>> -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core
>>>> BUILD=p
>>>> CAT=cat
>>>> CC=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/bin/intel64/icc
>>>> CCVERSION=11.1
>>>> CC_FRONT=
>>>> CDEBUG=-g $(addprefix $(CDEFINE),_DEBUG)
>>>> CDEFINE=-D
>>>> CFLAGS=-ftz -vec-report0 -DINT64 -DZLIB -openmp
>>>> CLEARSPEEDLIB=
>>>> CMPPINCLUDE=/bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install/include
>>>> COPT=-O2
>>>> COPT0=-O0
>>>> COPT1=-O1
>>>> COPT2=-O2
>>>> COPT3=-O3
>>>> CP=cp -p
>>>> CPROFILE=-p
>>>> CSCN=cscn
>>>> CSFLAGS=-O3 -I. --dynamic
>>>> CUDACC=
>>>> CUDACCVERSION=
>>>> CUDACDEBUG=-g $(addprefix $(CUDACDEFINE),_DEBUG)
>>>> CUDACDEFINE=-D
>>>> CUDACFLAGS=
>>>> CUDACOPT=
>>>> CUDACOPT0=-O0
>>>> CUDACOPT1=-O1
>>>> CUDACOPT2=-O2
>>>> CUDACOPT3=-O3
>>>> CUDACPROFILE=-p
>>>> CXX=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/bin/intel64/icpc
>>>> CXXFLAGS=$(CFLAGS)
>>>> DOXYGEN=/usr/bin/doxygen
>>>> ECHO=echo
>>>> EXPORT=export
>>>> F90FLAGS=
>>>> FC=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/bin/intel64/ifort
>>>> FCVERSION=11.1
>>>> FDEBUG=-g $(addprefix $(FDEFINE),_DEBUG)
>>>> FDEFINE=-D
>>>> FFLAGS=-i8 -r8 -pc64 -auto -Vaxlib -vec-report0 -cxxlib -openmp
>>>> FOPT=-O3
>>>> FOPT0=-O0
>>>> FOPT1=-O1
>>>> FOPT2=-O2
>>>> FOPT3=-O3
>>>> FPP=-fpp
>>>> FPROFILE=-p
>>>> FSTATIC=
>>>> FTCFLAGS=molpro unix unix-i8 Linux lapack mpp blas
>>>> HDF5INCLDE=
>>>> HDF5LIB=
>>>> HOSTFILE_FORMAT=
>>>> INSTBIN=/bwfs/scratch/temphomes/hd/f81/molpro2010.1-install/bin
>>>> INSTHTML=/bwfs/scratch/temphomes/hd/f81/molpro2010.1-install/html
>>>> INSTLIB=/bwfs/scratch/temphomes/hd/f81/molpro2010.1-install/lib/molprop_2010_1_Linux_x86_64_i8
>>>>
>>>>
>>>> INST_PL=0
>>>> INTEGER=8
>>>> LAPACKLIB=
>>>> LATEX2HTML=
>>>> LAUNCHER=/opt/bwgrid/mpi/mpiexec/0.84/bin/mpiexec -n %n %x
>>>> LD_ENV=/opt/bwgrid/compiler/intel/ct_4.0/Compiler/11.1/072/lib/intel64:/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t
>>>>
>>>>
>>>> LD_ENVNAME=LD_LIBRARY_PATH
>>>> LIBRARY_SUFFIX=a
>>>> LIBS=-lz
>>>> LIBS_FRONT=
>>>> LINKOPT=
>>>> LINKOPT_FRONT=
>>>> LN=ln -s
>>>> MACROS=MOLPRO MOLPRO_ifort MOLPRO_f2003 GA_TOOLS GA_ GA_VERSION_GE_5
>>>> _I8_ BLAS_INT=8 LAPACK_INT=8 MOLPRO_FORCE_VECTOR MOLPRO_NEXTSCALAR
>>>> MOLPRO_NO_RECURRENCE MOLPRO_NOVECTOR MOLPRO_SHORTLOOP _MOLCAS_MPP_
>>>> MAKEDEPEND_OPTIONS=
>>>> MAKEINDEX=/usr/bin/makeindex
>>>> MAPLE=
>>>> MKDIR=mkdir -p
>>>> MODULE_FLAG=-I
>>>> MODULE_SUFFIX=mod
>>>> MPILIB=-L/opt/bwgrid/compiler/intel/ct_4.0/impi/4.0.0.028/intel64/lib
>>>> -lmpi -lmpigf -lmpigi -lpthread -L/usr/lib64 -libverbs -libcommon
>>>> -libumad -lpthread
>>>> MPPLIB=-L/bwfs/scratch/temphomes/hd/f81/ga-5-0-2-install/lib/LINUX64
>>>> -lga -larmci
>>>> OBJECT_SUFFIX=o
>>>> OPT0=copyc6.f
>>>> OPT1=nevpt2_optrpc.f explicit_util.f artwo2.f drv2el_l3ext_lmp2g.f
>>>> drv2el_l3ext_cen.f rmp2_f12_drv2.f90 ri_lmp2g.f df_llmp2.f readdump.f
>>>> OPT2=integrals.f90 RL4gen1.f basis_integral_shells.f
>>>> OPT3=
>>>> PAPER=a4paper
>>>> PARSE=parse-Linux-x86_64-i8.o
>>>> PDFLATEX=/usr/bin/pdflatex -halt-on-error
>>>> PNAME=molprop_2010_1_Linux_x86_64_i8
>>>> PTSIZE=11
>>>> RANLIB=ranlib
>>>> RM=rm -rf
>>>> SHELL=/bin/sh
>>>> STRIP=strip
>>>> SUFFIXES=f F f90 F90 c cpp
>>>> TAR=tar -cf
>>>> UNTAR=tar -xf
>>>> VERBOSE=@
>>>> VERSION=2010.1XSD=/usr/bin/xmllint --noout --schema
>>>> XSLT=/usr/bin/xsltproc
>>>> YACC=bison -b y
>>>>
>>>> .SUFFIXES:
>>>> MAKEFLAGS+=-r
>>>> ifneq ($(LD_ENVNAME),)
>>>> $(LD_ENVNAME):=$(LD_ENV):$($(LD_ENVNAME))
>>>> endif
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Best regards,
>>>> Evgeniy
>>>>
>>>> Manhui Wang wrote:
>>>>> Dear Evgeniy,
>>>>>
>>>>> I have not checked your output carefully, which shows 800MWord (6.4GB
>>>>> per process):
>>>>>
>>>>> Variable memory set to  800000000 words,  buffer space   230000 words
>>>>>
>>>>> I run the job  with 32 processes on 4 nodes, and it might crash
>>>>> eventually due to lack of memory you mentioned.
>>>>>
>>>>> Could you please update your progress with GA? On my side, I will look
>>>>> into my code related to helper. It might exist a bug which is exposed
>>>>> when very large global data structure is used.
>>>>>
>>>>> Best wishes,
>>>>> Manhui
>>>>>
>>>>>
>>>>>
>>>>> Evgeniy Gromov wrote:
>>>>>> Dear Manhui,
>>>>>>
>>>>>> Many thanks for your help with the compilation problems and
>>>>>> with the problem in the large eom-ccsd calculation.
>>>>>> Sorry I forgot to tell you about the memory. I usually specify
>>>>>> the memory in the command line using -m option. For that large
>>>>>> eom-ccsd job I used 800m (-m800m) because 200m was not enough for
>>>>>> the CCSD step. As I wrote the job passed the CCSD and crashed
>>>>>> in the beginning of the EOM-CCSD. Actually, are you running it
>>>>>> on 1 CPU? I haven't tried it to run on 1 CPU as it will likely
>>>>>> take ages cause many MOs. Well I am going to rerun it using
>>>>>> a new binary compiled with GA.
>>>>>>
>>>>>> Best wishes,
>>>>>> Evgeniy
>>>>>>
>>>>>>
>>>>>> Manhui Wang wrote:
>>>>>>> Dear Evgeniy,
>>>>>>>
>>>>>>> I can't see the error you mentioned with MPI-2 Molpro2010.1 PL20 so
>>>>>>> far.
>>>>>>> I tried your input without any modification first, and it failed
>>>>>>> immediately because you have not set memory in input (the default is
>>>>>>> only 8MWord). I have added
>>>>>>>
>>>>>>> memory,200,m
>>>>>>>
>>>>>>> in input, it appears to work fine so far (still running).  Could you
>>>>>>> please try the new input on your side?
>>>>>>>
>>>>>>> Best wishes,
>>>>>>> Manhui
>>>>>>>
>>>>>>>
>>>>>>> Evgeniy Gromov wrote:
>>>>>>>> Manhui,
>>>>>>>>
>>>>>>>> I attached wrong files in the previous email. Here are the correct
>>>>>>>> input and output files. In addition I removed all the MO vectors in
>>>>>>>> the output to make the size smaller.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Evgeniy
>>>>>>>>
>>>>>>>> Manhui Wang wrote:
>>>>>>>>> Dear Evgeniy,
>>>>>>>>>
>>>>>>>>> Could you please provide the inputs/outputs? I will see whether
>>>>>>>>> I can
>>>>>>>>> reproduce the problem firstly.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Manhui
>>>>>>>>>
>>>>>>>>> Evgeniy Gromov wrote:
>>>>>>>>>> Dear Molpro Community,
>>>>>>>>>>
>>>>>>>>>> I faced problems when running a large eom-ccsd calculation (466
>>>>>>>>>> MOs,
>>>>>>>>>> no symmetry) with molpro2010.1 in parallel (MPI-2). The
>>>>>>>>>> calculation
>>>>>>>>>> crashed after the CCSD (ground state) step had been done. No error
>>>>>>>>>> message was present in the output. In the err file there is the
>>>>>>>>>> following diagnostic (though unclear to me):
>>>>>>>>>>
>>>>>>>>>> ERROR in twosided_helpga_locate_server: over range!
>>>>>>>>>> ilo=1,ihigh=148114575
>>>>>>>>>>
>>>>>>>>>> Any idea what can be the problem.
>>>>>>>>>>
>>>>>>>>>> Well, it seems to relate to helper process. I suspect that in the
>>>>>>>>>> case
>>>>>>>>>> of GA it would work, although there might be some other problems.
>>>>>>>>>> Just in case the same run with 296 MO (another basis set) ran
>>>>>>>>>> fine.
>>>>>>>>>>
>>>>>>>>>> Also, I would like to thank Manhui for the response to my previous
>>>>>>>>>> post:
>>>>>>>>>> "Is there always a helper process in parallel calculations with
>>>>>>>>>> molpro2010.1 ?"
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Evgeniy
>>>>>>>>>>
>>>>>>>>>>
>>
> 

-- 
_______________________________________
Dr. Evgeniy Gromov
Theoretische Chemie
Physikalisch-Chemisches Institut
Im Neuenheimer Feld 229
D-69120 Heidelberg
Germany

Telefon: +49/(0)6221/545263
Fax: +49/(0)6221/545221
E-mail: evgeniy at pci.uni-heidelberg.de
_______________________________________