[molpro-user] Inconsistent parralel behavior

[Fogarty, Richard M]

Dear Molpro users,

I'm having trouble running casscf calculations in parralel, though simpler calculations seem to run somewhat ok. Specifically i get some speedup for Hartree-Fock calculations with multiple cores (compared to 1 core), but no significant speedup for casscf calculations (or caspt2).

For example here are some timings (in seconds) i get for a test HF and test-casscf job:
CPUs Real time (HF)        Real time (CASSCF)
1 6712.52 2360.99
4 5870.76 1816.12
8 1424.63 2032.02
12 1019.93 1809.47
16 938.01 2318.11
20 813.71 3847.01

So the HF job definitely gets at least some speedup, while the casscf times seem essentially random/unaffected by adding more cores. These results come from running the pre-compiled binary version on a hpc with PBS.

Any help would be greatly appreciated

Thanks

Richard



Relevant files are below:

The casscf input file looks like this:

"memory,627,m !627 megawords of memory (1MW~8MB)
FILE,2,hf_ccpvtz_12c.wfu
GDIRECT
symmetry,nosym
basis={
default=vtz
}
geometry={ OMITTED}
n_frozen = 10
n_closed = 29
n_occ = 39
n_trip_states = 3
n_sing_states = 2
{multi;
occ,n_occ
closed,n_closed
frozen,n_frozen
wf,spin=0;state,n_sing_states;
wf,spin=2;state,n_trip_states}"

The PBS submit script is as follows

# batch processing commands
#PBS -l walltime=59:59:00
#PBS -lselect=1:ncpus=12:mpiprocs=12:ompthreads=1:mem=67000MB:tmpspace=400000MB
#PBS -j oe
#PBS -q pqmaterials
#PBS -m n

#Get the relevant restart file in correct place
rs_line=$(grep 'FILE,2' $PBS_O_WORKDIR/${in}) #Line with wfn file
rs_line=${rs_line%\!*} #Remove the comment
full_wfu_name=${rs_line##*,} #Get substring containing rs file name
full_wfu_name=${full_wfu_name:0:32}
full_wfu_name="${full_wfu_name,,}"
cp $PBS_O_WORKDIR/$full_wfu_name $TMPDIR/.
cp $PBS_O_WORKDIR/${in} $TMPDIR/.

#Run molpro
cust_error_file="${in/.com/.error}"
pbsexec /home/rf614/MolPro/bin/molpro -t 1 -n ${core} -d $TMPDIR -W $TMPDIR -v ${in} >> $TMPDIR/$cust_error_file

#cp files back
cp $TMPDIR/*.out /$PBS_O_WORKDIR/.
cp $TMPDIR/*.xml /$PBS_O_WORKDIR/.
cp $TMPDIR/$cust_error_file /$PBS_O_WORKDIR/.


Molpro outputs the following to $cust_error_file (This is for the 12-core case):

 # PARALLEL mode, HOST=cx1-130-5-15
 nodelist=12
 first   =12
 second  =
 third   =
 HOSTFILE_FORMAT: $hostname

cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk
cx1-130-5-15.cx1.hpc.ic.ac.uk

        LD_LIBRARY_PATH=''
 export AIXTHREAD_SCOPE='s'
 export MOLPRO_PREFIX='/export131/home/rf614/MolPro/install'
 export MP_NODES='0'
 export MP_PROCS='12'
        MP_TASKS_PER_NODE=''
 export MOLPRO_NOARG='1'
 export MOLPRO_OPTIONS=' -d /var/tmp/pbs.398951.cx1 -W /var/tmp/pbs.398951.cx1 -v /work/rf614/Post_Doc/molpro_tests/omp_tests/casscf_runs/casscf_part_only/test_casscf.com -t 1'
        MOLPRO_OPTIONS_FILE=''
        MPI_MAX_CLUSTER_SIZE=''
        MV2_ENABLE_AFFINITY=''
 export RT_GRQ='ON'
 export TMPDIR='/var/tmp/pbs.398951.cx1'
 export XLSMPOPTS='parthds=1'
/export131/home/rf614/MolPro/install/bin/mpiexec.hydra -machinefile /var/tmp/pbs.398951.cx1/procgrp.33961 -np 12 /export131/home/rf614/MolPro/install/bin/molpro.exe  -d /var/tmp/pbs.398951.cx1 -W /var/tmp/pbs.398951.cx1 -v /work/rf614/Post_Doc/molpro_tests/omp_tests/casscf_runs/casscf_part_only/test_casscf.com -t 1


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.molpro.net/pipermail/molpro-user/attachments/20171029/8c1f78bf/attachment-0001.html>