I eventually got everything to work as well, except for the "factory" binaries. The grid engine on our cluster is MOAB (commercial version of Torque+MAUI), but SGE should work as well. Below are my installation notes. Obviously, directory locations etc. are specific to our HPC cluster. Throughout, the compilers and math library used are Intel 11.1 and MKL 10, respectively.<div>
<br></div><div>Jan</div><div><br><div><div><div style="font-family: Arial; "><b>To use vanilla TCGMSG</b></div><div style="font-family: Arial; "><b><br></b></div><div style="font-family: Arial; ">(1) download GA 4.3.2. Install inside MOLPRO directory and (on one line)</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">make TARGET=LINUX64 FC=ifort CC=icc </div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">IMPORTANT: using TARGET=LINUX will generate executables that seem to work OK for CCSD(T) but produce garbage for MRCI</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">(2)</div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">/configure -icc -ifort -blaspath /usr/local/intel/mkl/<a href="http://10.2.2.025/lib/em64t">10.2.2.025/lib/em64t</a> -m<b><span class="Apple-style-span" style="font-weight: normal; ">pp -mppbase /home/comartin/sandbox/molpro2010.1-ga-tcgmsg/</span></b>ga-4-3-2<b><span class="Apple-style-span" style="font-weight: normal; "> </span><span class="Apple-style-span" style="font-weight: normal; ">-instroot /home/comartin/apps/molpro2010.1-ifort-11.059-mkl-tcgmsg</span></b></div>
<div style="font-family: Arial; "><b><span class="Apple-style-span" style="font-weight: normal; "><br></span></b></div><div style="font-family: Arial; "><b><span class="Apple-style-span" style="font-weight: normal; ">make -j16 </span></b></div>
<div style="font-family: Arial; "><b><span class="Apple-style-span" style="font-weight: normal; "><br></span></b></div><div style="font-family: Arial; "><b><span class="Apple-style-span" style="font-weight: normal; ">(3) the resulting executables work fine interactively but not on PBS thanks to the dual network (Gbit and InfiniBand). The following will make it work: edit bin/molpro and look for these two lines</span></b></div>
<div style="font-family: Arial; "><b><span class="Apple-style-span" style="font-weight: normal; "><br></span></b></div><div style="font-family: Arial; "><b><span class="Apple-style-span" style="font-weight: normal; "> PROCGRP="$hostfile"<div>
fi</div><div><br></div><div>then append the following block after them</div><div>#</div><div>echo JMdebug dirty hack for local two networks</div><div>sed 's/-ib0//g' <$hostfile >/tmp/editedhostfile.$$</div>
<div>cat /tmp/editedhostfile.$$ >$hostfile</div><div>echo hostfile is</div><div>cat $hostfile</div><div>echo JMdebug end hostfile</div></span></b></div><div style="font-family: Arial; ">#</div><div style="font-family: Arial; ">
it should now work also in batch. Note that communication via sockets may be suboptimal</div><div style="font-family: Arial; "><b><br></b></div><div style="font-family: Arial; "><b>To use TCGMSG with IMPI (the Intel MPI shipping with the compilers):</b></div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">(1) download GA 4.3.2. Install inside MOLPRO directory and (on one line)</div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">
make TARGET=LINUX64 FC=ifort CC=icc MPI_LIB=/usr/local/intel/impi/<a href="http://3.2.2.006/lib64">3.2.2.006/lib64</a> MPI_INCLUDE=/usr/local/intel/impi/<a href="http://3.2.2.006/include64">3.2.2.006/include64</a> USE_MPI=yes</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">IMPORTANT: using TARGET=LINUX will generate executables that work OK for CCSD(T) but generate garbage for MRCI</div><div style="font-family: Arial; ">
<br></div><div style="font-family: Arial; ">(2) then (assuming you are in /home/comartin/sandbox/molpro2010.1-try3)</div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">set path=(/usr/local/intel/impi/<a href="http://3.2.2.006/bin64">3.2.2.006/bin64</a> $path)</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">./configure -icc -ifort -blaspath /usr/local/intel/mkl/<a href="http://10.2.2.025/lib/em64t">10.2.2.025/lib/em64t</a> -mpp -mppbase /home/comartin/sandbox/molpro2010.1-try3/ga-4-3-2 -instroot /home/comartin/apps/molpro2010.1-ifort-11.059-mkl-tcgmsg</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">make -j16</div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">(3) the following script should work for a test job</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">#!/bin/tcsh<div>cd /home/comartin/sandbox/molpro2010.1-try3</div><div>echo RUNNING ON HOST $HOST</div><div>#setenv TCGRSH /usr/bin/ssh</div><div>
set path=(/usr/local/intel/impi/<a href="http://3.2.2.006/bin64">3.2.2.006/bin64</a> $path)</div><div>setenv LD_LIBRARY_PATH /usr/local/intel/impi/<a href="http://3.2.2.006/lib64:$LD_LIBRARY_PATH">3.2.2.006/lib64:$LD_LIBRARY_PATH</a></div>
<div>/usr/local/intel/impi/<a href="http://3.2.2.006/bin64/mpd">3.2.2.006/bin64/mpd</a> &</div><div>bin/molpro -v -n 4 -m 64M -d /scratch/$USER <a href="http://n2test.com/">n2test.com</a></div></div><div style="font-family: Arial; ">
<br></div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; "><b>To use TCGMSG with OpenMPI:</b></div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">for ga-4-3-2:</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">make TARGET=LINUX64 FC=mpif77 CC=mpicc MPI_LIB=/usr/local/openmpi/lib MPI_BIN=/usr/local/openmpi/bin USE_MPI=yes</div><div style="font-family: Arial; ">
<br></div><div style="font-family: Arial; ">for molpro:</div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">./configure -icc -ifort -blaspath /usr/local/intel/mkl/<a href="http://10.2.2.025/lib/em64t">10.2.2.025/lib/em64t</a> -mpp -mppbase /home/comartin/sandbox/molpro2010.1-ga-openmpi/ga-4-3-2 -instroot /home/comartin/apps/molpro2010.1-ifort-11.059-mkl-ga-openmpi</div>
<div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">in order to run on some systems you may need to adjust a system parameter:</div><div style="font-family: Arial; "><br></div><div style="font-family: Arial; ">
<a href="http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages">http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages</a><br></div><div style="font-family: Arial; font-size: medium; "><br></div><div class="gmail_quote">
On Tue, Sep 28, 2010 at 1:25 PM, Reuti <span dir="ltr"><<a href="mailto:reuti@staff.uni-marburg.de">reuti@staff.uni-marburg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi,<br>
<br>
Am 27.09.2010 um 22:12 schrieb Tanja van Mourik:<br>
<div class="im"><br>
> Hi Jan and Manhui,<br>
><br>
>>> I agree it's strange that it works interactively but not through PBS.<br>
>>> Perhaps to put this rsh vs. ssh stuff to rest you could try the following:<br>
>>><br>
>>> 1) put the following in your .cshrc file so that it is set on all your<br>
>>> nodes: setenv RSHCOMMAND /usr/bin/ssh<br>
>>><br>
>>> 2) if that doesn't help, just make a soft link (on all nodes) that<br>
>>> points /usr/bin/rsh to /usr/bin/ssh<br>
>>><br>
>>> Manhui: why is rsh the default and not ssh for the parallel binaries?<br>
>>> No one uses rsh any more.....<br>
>> The current parallel molpro binaries are still built using GA with<br>
>> TCGMSG, which is very portable, and the default connection in TCGMSG is<br>
>> rsh. We are considering using GA with TCGMSG-MPI in future versions.<br>
>><br>
>> After discussion with Jan, he has found the problem lies in that the HPC<br>
>> cluster has two kinds of networks, and is looking for a solution.<br>
><br>
> Has a solution been found for this problem? I encounter something very similar<br>
> with SGE. We compiled molpro2009 and now installed the binary of molpro2010,<br>
> but both versions give the following error message when trying to run via SGE:<br>
<br>
</div>I run both self compiled versions w/o problems inside SGE. But in all cases I used Open MPI (--with-sge) as transport (either native or for GA's TCGMSG). I used the Portland compilers though.<br>
<font color="#888888"><br>
-- Reuti<br>
</font><div><div></div><div class="h5"><br>
<br>
> forrtl: severe (64): input conversion error, unit -5, file Internal Formatted<br>
> Read<br>
><br>
> It works when running interactively on the same node with the same command as<br>
> used in the job. Setting "setenv RSHCOMMAND /usr/bin/ssh" and "setenv TCGRSH<br>
> /usr/bin/ssh" in .cshrc or in the job script does not make a difference. This<br>
> should anyway not be the problem, as I can rsh between nodes.<br>
><br>
> Best wishes,<br>
><br>
> Tanja<br>
> --<br>
> =================================================================<br>
> Tanja van Mourik<br>
> Senior Lecturer in Chemistry<br>
> School of Chemistry, University of St. Andrews<br>
> North Haugh, St. Andrews, Fife KY16 9ST, Scotland (UK)<br>
><br>
> email: <a href="mailto:tanja.vanmourik@st-andrews.ac.uk">tanja.vanmourik@st-andrews.ac.uk</a><br>
> web: <a href="http://chemistry.st-and.ac.uk/staffmember.php?id=tvm" target="_blank">http://chemistry.st-and.ac.uk/staffmember.php?id=tvm</a><br>
><br>
> The University of St Andrews is a charity registered<br>
> in Scotland: No SC013532<br>
> =================================================================<br>
>><br>
><br>
><br>
> ------------------------------------------------------------------<br>
> University of St Andrews Webmail: <a href="https://webmail.st-andrews.ac.uk" target="_blank">https://webmail.st-andrews.ac.uk</a><br>
><br>
> _______________________________________________<br>
> Molpro-user mailing list<br>
> <a href="mailto:Molpro-user@molpro.net">Molpro-user@molpro.net</a><br>
> <a href="http://www.molpro.net/mailman/listinfo/molpro-user" target="_blank">http://www.molpro.net/mailman/listinfo/molpro-user</a><br>
<br>
_______________________________________________<br>
Molpro-user mailing list<br>
<a href="mailto:Molpro-user@molpro.net">Molpro-user@molpro.net</a><br>
<a href="http://www.molpro.net/mailman/listinfo/molpro-user" target="_blank">http://www.molpro.net/mailman/listinfo/molpro-user</a><br>
</div></div></blockquote></div><br><br>
</div></div></div>