<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p></p>
<div>Dear Molpro users,</div>
<div><br>
</div>
<div>I'm having trouble running casscf calculations in parralel, though simpler calculations seem to run somewhat ok. Specifically i get some speedup for Hartree-Fock calculations with multiple cores (compared to 1 core), but no significant speedup for casscf
calculations (or caspt2). </div>
<div><br>
</div>
<div>For example here are some timings (in seconds) i get for a test HF and test-casscf job:</div>
<div>CPUs<span class="Apple-tab-span" style="white-space:pre"> </span>Real time (HF)<span class="Apple-tab-span" style="white-space:pre">
</span> <span style="font-family: Calibri, Helvetica, sans-serif;"> </span>Real time (CASSCF)</div>
<div>1<span class="Apple-tab-span" style="white-space:pre"> </span>6712.52<span class="Apple-tab-span" style="white-space:pre">
</span>2360.99<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div>4<span class="Apple-tab-span" style="white-space:pre"> </span>5870.76<span class="Apple-tab-span" style="white-space:pre">
</span>1816.12</div>
<div>8<span class="Apple-tab-span" style="white-space:pre"> </span>1424.63<span class="Apple-tab-span" style="white-space:pre">
</span>2032.02</div>
<div>12<span class="Apple-tab-span" style="white-space:pre"> </span>1019.93<span class="Apple-tab-span" style="white-space:pre">
</span>1809.47</div>
<div>16<span class="Apple-tab-span" style="white-space:pre"> </span>938.01<span class="Apple-tab-span" style="white-space:pre">
</span>2318.11</div>
<div>20<span class="Apple-tab-span" style="white-space:pre"> </span>813.71<span class="Apple-tab-span" style="white-space:pre">
</span>3847.01</div>
<div><br>
</div>
<div>So the HF job definitely gets at least some speedup, while the casscf times seem essentially random/unaffected by adding more cores. These results come from running the pre-compiled binary version on a hpc with PBS.</div>
<div><br>
</div>
<div>Any help would be greatly appreciated</div>
<div><br>
</div>
<div>Thanks</div>
<div><br>
</div>
<div>Richard</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Relevant files are below:</div>
<div><br>
</div>
<div>The casscf input file looks like this:</div>
<div><br>
</div>
<div>"memory,627,m !627 megawords of memory (1MW~8MB)</div>
<div>FILE,2,hf_ccpvtz_12c.wfu </div>
<div>GDIRECT</div>
<div>symmetry,nosym</div>
<div>basis={</div>
<div>default=vtz</div>
<div>}</div>
<div>geometry={ OMITTED}</div>
<div>n_frozen = 10 </div>
<div>n_closed = 29 </div>
<div>n_occ = 39 </div>
<div>n_trip_states = 3 </div>
<div>n_sing_states = 2 </div>
<div>{multi; </div>
<div>occ,n_occ</div>
<div>closed,n_closed</div>
<div>frozen,n_frozen</div>
<div>wf,spin=0;state,n_sing_states;</div>
<div>wf,spin=2;state,n_trip_states}"</div>
<div><br>
</div>
<div>The PBS submit script is as follows</div>
<div><br>
</div>
<div># batch processing commands</div>
<div>#PBS -l walltime=59:59:00</div>
<div>#PBS -lselect=1:ncpus=12:mpiprocs=12:ompthreads=1:mem=67000MB:tmpspace=400000MB</div>
<div>#PBS -j oe</div>
<div>#PBS -q pqmaterials</div>
<div>#PBS -m n</div>
<div><br>
</div>
<div>#Get the relevant restart file in correct place</div>
<div>rs_line=$(grep 'FILE,2' $PBS_O_WORKDIR/${in}) #Line with wfn file</div>
<div>rs_line=${rs_line%\!*} #Remove the comment</div>
<div>full_wfu_name=${rs_line##*,} #Get substring containing rs file name</div>
<div>full_wfu_name=${full_wfu_name:0:32} </div>
<div>full_wfu_name="${full_wfu_name,,}" </div>
<div>cp $PBS_O_WORKDIR/$full_wfu_name $TMPDIR/.</div>
<div>cp $PBS_O_WORKDIR/${in} $TMPDIR/.</div>
<div><br>
</div>
<div>#Run molpro</div>
<div>cust_error_file="${in/.com/.error}"</div>
<div>pbsexec /home/rf614/MolPro/bin/molpro -t 1 -n ${core} -d $TMPDIR -W $TMPDIR -v ${in} >> $TMPDIR/$cust_error_file</div>
<div><br>
</div>
<div>#cp files back</div>
<div>cp $TMPDIR/*.out /$PBS_O_WORKDIR/.</div>
<div>cp $TMPDIR/*.xml /$PBS_O_WORKDIR/.</div>
<div>cp $TMPDIR/$cust_error_file /$PBS_O_WORKDIR/.</div>
<div><br>
</div>
<div><br>
</div>
<div>Molpro outputs the following to $cust_error_file (This is for the 12-core case):</div>
<div><br>
</div>
<div> # PARALLEL mode, HOST=cx1-130-5-15</div>
<div> nodelist=12</div>
<div> first =12</div>
<div> second =</div>
<div> third =</div>
<div> HOSTFILE_FORMAT: $hostname</div>
<div><br>
</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div>cx1-130-5-15.cx1.hpc.ic.ac.uk</div>
<div><br>
</div>
<div> LD_LIBRARY_PATH=''</div>
<div> export AIXTHREAD_SCOPE='s'</div>
<div> export MOLPRO_PREFIX='/export131/home/rf614/MolPro/install'</div>
<div> export MP_NODES='0'</div>
<div> export MP_PROCS='12'</div>
<div> MP_TASKS_PER_NODE=''</div>
<div> export MOLPRO_NOARG='1'</div>
<div> export MOLPRO_OPTIONS=' -d /var/tmp/pbs.398951.cx1 -W /var/tmp/pbs.398951.cx1 -v /work/rf614/Post_Doc/molpro_tests/omp_tests/casscf_runs/casscf_part_only/test_casscf.com -t 1'</div>
<div> MOLPRO_OPTIONS_FILE=''</div>
<div> MPI_MAX_CLUSTER_SIZE=''</div>
<div> MV2_ENABLE_AFFINITY=''</div>
<div> export RT_GRQ='ON'</div>
<div> export TMPDIR='/var/tmp/pbs.398951.cx1'</div>
<div> export XLSMPOPTS='parthds=1'</div>
<div>/export131/home/rf614/MolPro/install/bin/mpiexec.hydra -machinefile /var/tmp/pbs.398951.cx1/procgrp.33961 -np 12 /export131/home/rf614/MolPro/install/bin/molpro.exe -d /var/tmp/pbs.398951.cx1 -W /var/tmp/pbs.398951.cx1 -v /work/rf614/Post_Doc/molpro_tests/omp_tests/casscf_runs/casscf_part_only/test_casscf.com
-t 1</div>
<div><br>
</div>
<br>
<p></p>
</div>
</body>
</html>