<div dir="ltr"><div><div><div>Hello,<br></div>Signal 9 usually indicates to us that the job was killed from exceeding memory and the logs indeed show:<br>12/19/2017 20:04:20;0008;pbs_python;Job;2169493.r-man2;Cgroup memory limit exceeded: Killed process <br>24728 (molpro.exe) total-vm:17508688kB, anon-rss:9439868kB, file-rss:4816kB, shmem-rss:264kB<br></div>This indicates a sudden memory hike that killed the job before it had a chance to get logged by PBS.<br></div>Rika<br><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 19 December 2017 at 21:10, Seth Olsen <span dir="ltr"><<a href="mailto:seth.olsen@uq.edu.au" target="_blank">seth.olsen@uq.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">



<div style="word-wrap:break-word">
Hi Molpro-User,
<div><br>
</div>
<div>I’ve been having a job fail during orbital localization.  It is a CASSCF (8 electron in 8 orbital) job.  The output ends abruptly:</div>
<div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"> *****************************<wbr>******************************<wbr>******************************<wbr>******************************<wbr>***********</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"> Program * Orbital Localization         Authors:  W. Meyer, H.-J. Werner</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"> Pipek-Mezey Localization</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"> Molecular orbitals read from record     2141.2  Type=MCSCF/NATURAL  </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"> Density matrix read from record         2141.2  Type=MCSCF/CHARGE (state averaged)</span></div>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>…but the standard output has a little more description</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">MPI_ABORT was invoked on rank 1 in communicator MPI COMMUNICATOR 4 DUP FROM 0 </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">with errorcode 15.</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">You may or may not see output from other processes, depending on</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">exactly when Open MPI kills them.</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">0:Terminate signal was sent, status=: 15</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">(rank:0 hostname:r2398 pid:24716):ARMCI DASSERT fail. src/common/signaltrap.c:<wbr>SigTermHandler():477 cond:0</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">mpirun noticed that process rank 7 with PID 24730 on node r2398 exited on signal 9 (Killed).</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>--------------</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">==============================<wbr>==============================<wbr>==========================</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">                  Resource Usage on 2017-12-19 20:04:32:</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   Job Id:             2169493.r-man2</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   Project:            tv58</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   Exit Status:        0</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   Service Units:      0.55</span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   NCPUs Requested:    8                      NCPUs Used: 8               </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">                                           CPU Time Used: 00:24:10                                   </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   Memory Requested:   48.0GB                Memory Used: 31.96GB         </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   Walltime requested: 03:00:00            Walltime Used: 00:04:09        </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">   JobFS requested:    150.0GB                JobFS used: 941.47MB        </span></div>
<div style="margin:0px;font-size:10px;line-height:normal;font-family:Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">==============================<wbr>==============================<wbr>==========================</span></div>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">Any ideas?  Anyone seen this before?  </span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>Many Thanks,</div>
<div>Seth<br>
<div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>===========================
<div>Seth Olsen, PhD.</div>
<div>Honorary Fellow</div>
<div>School of Mathematics & Physics</div>
<div>The University of Queensland</div>
<div>QLD 4072  Australia</div>
<div>Ph: <a href="tel:+61%207%203365%202816" value="+61733652816" target="_blank">+61 7 3365 2816</a></div>
<div>===========================</div>
<div>A PGP public key for this address has been uploaded to the key servers.</div>
</div>
</div>
</div>
<br>
</div>
</div>

<br>______________________________<wbr>_________________<br>
Molpro-user mailing list<br>
<a href="mailto:Molpro-user@molpro.net">Molpro-user@molpro.net</a><br>
<a href="http://www.molpro.net/mailman/listinfo/molpro-user" rel="noreferrer" target="_blank">http://www.molpro.net/mailman/<wbr>listinfo/molpro-user</a><br></blockquote></div><br></div>