[molpro-user] ARMCI test errors after compiling with GA/MPICH2, gfortran, ATLAS on Ubuntu 11.04

[Gregory Magoon]

Thanks Andy,
Unfortunately, when I tried approach #1, it gave the same run-time failure
pattern in the tests and it was actually slower (at least for the big_lattice
results I have been using as benchmark). I have gone back to using the regular
blas and lapack ubuntu packages, which seems to work without error, though it
seems to be a factor of 2+ slower.

I've placed my installation notes for the failed attempt with 8-byte integer
atlas below in case you can spot that I'm doing anything wrong.
Greg


#get dependencies

sudo apt-get install mpich2
sudo apt-get install curl
sudo apt-get install gfortran
sudo apt-get install zlib1g-dev
sudo apt-get install libcr-dev

#disable CPU throttling (http://ubuntuforums.org/showthread.php?t=611084)
sudo /usr/bin/cpufreq-selector -g performance

#install LAPACK (cf section 8 of ATLAS install guide:
http://math-atlas.sourceforge.net/atlas_install/atlas_install.html#SECTION00090000000000000000
)
(in lapack directory)
cp /INSTALL/make.gfortran.inc make.inc
#edit make.inc to have the following:
FORTRAN  = gfortran
OPTS     = -O -fPIC -m64
DRVOPTS  = $(OPTS)
NOOPT    = -O0 -fPIC -m64
LOADER   = gfortran
LOADOPTS = $(OPTS)
TIMER    = INT_ETIME

make lib


#from atlas build directory:
../configure -b 64 -A 0 -D c -DWALL -Fa alg -fPIC -Si archdef 0
--incdir=/usr/include/ATLAS3.8.4 --libdir=/usr/lib64/ATLAS3.8.4
--with-netlib-lapack=/home/user/lapack-3.3.1/lapack_LINUX.a
make
cd (build)/lib; make shared; make ptshared
make check; make ptcheck
make time
sudo make install
#manually copy dynamic libraries
sudo cp lib/*.so /usr/lib64/ATLAS3.8.4/
sudo chmod 0644 /usr/lib64/ATLAS3.8.4/*.so

(remove existing lapack, blas library packages not needed: sudo apt-get 
remove)

#point to the atlas libraries
cd /usr/lib64
sudo ln -s /usr/lib64/ATLAS3.8.4/liblapack.a liblapack.a
sudo ln -s /usr/lib64/ATLAS3.8.4/liblapack.so liblapack.so
sudo ln -s /usr/lib64/ATLAS3.8.4/libcblas.a libcblas.a
sudo ln -s /usr/lib64/ATLAS3.8.4/libcblas.so libcblas.so
sudo ln -s /usr/lib64/ATLAS3.8.4/libf77blas.a libf77blas.a
sudo ln -s /usr/lib64/ATLAS3.8.4/libf77blas.so libf77blas.so
sudo ln -s /usr/lib64/ATLAS3.8.4/libatlas.a libatlas.a
sudo ln -s /usr/lib64/ATLAS3.8.4/libatlas.so libatlas.so
sudo ln -s /usr/lib64/ATLAS3.8.4/libptcblas.a libptcblas.a
sudo ln -s /usr/lib64/ATLAS3.8.4/libptcblas.so libptcblas.so
sudo ln -s /usr/lib64/ATLAS3.8.4/libptf77blas.a libptf77blas.a
sudo ln -s /usr/lib64/ATLAS3.8.4/libptf77blas.so libptf77blas.so

#compile Global arrays **with LAPACK and 8-sized BLAS libraries**
./configure TARGET=LINUX64 FC=gfortran --with-mpi='-I/usr/include/mpich2
-L/usr/lib64 -lmpichf90 -lmpich -lpthread -lmpl' --with-blas8='-L/usr/lib64
-lf77blas -lcblas -latlas' --with-lapack='-L/usr/lib64 -llapack'
make
make check
mpirun ./global/testing/test.x

#(as root)
sudo make install
#will be installed in /usr/local/bin and /usr/local/lib
copy config.log to /usr/local/config.log

#compile molpro
./configure -batch -gcc -gfortran -mpp -mppbase /usr/local -instroot
/usr/local/molpro2010.1
make

#tuning, testing
make patch #enter username, password to download token
make tuning
nohup make MOLPRO_OPTIONS=-n8 test > test052511e.txt &
#***original failure pattern, very slow; timing for big_lattice: 436/455

Quoting Andy May <MayAJ1 at cardiff.ac.uk>:

> Greg,
>
> Apologies for the delay, I was away last week.
>
> The problem with Molpro/ATLAS was a compile time problem. Essentially 
> Molpro should be able to be built with just the ATLAS routines, and 
> should automatically build the missing LAPACK routines, thus avoiding 
> the user to do this manually. This is now fixed in our repository, 
> and will be pushed out with some other fixes after testing. (Some 
> LAPACK routines not present in ATLAS were also not being compiled by 
> Molpro).
>
> I suspect the problem is that for ATLAS we assume it has been built 
> with 8-byte integers on 64-bit systems, and probably the ubuntu 
> binaries are built with 4-byte integers.
>
> Also Molpro configure has a dilemma when it finds libatlas and then 
> liblapack because the name 'liblapack' is rather generic. We assume 
> that liblapack is probably an ATLAS LAPACK library since libatlas is 
> present, but it is perfectly possible liblapack could be a system one 
> (on openSUSE for example liblapack is simply the netlib routines). So 
> the fact your liblapack is a full LAPACK library has been ignored, 
> and Molpro will have compiled the non-ATLAS routines internally (with 
> 8-byte integers), which will add to the confusion.
>
> To use ATLAS you could either:
>
> 1. Compile ATLAS by hand with 8-byte integers (this will also give 
> you optimal parameters for your hardware). Then build GA again using 
> --with-blas8 option, and then finally build Molpro. This will then 
> give you everything with 8-byte integers.
>
> 2. Proceed as before, but after configure edit CONFIG and change the 
> values of BLAS_INT and LAPACK_INT in the MACROS variable. I think 
> this should work, but haven't tried it.
>
> Best wishes,
>
> Andy
>
> On 23/05/11 18:23, Gregory Magoon wrote:
>> I was able to narrow this down to an issue with ATLAS. Andy, could you
>> elaborate
>> on your previous comments regarding Molpro not working with the
>> unmodified ATLAS
>> libraries? Is this a compile-time or run-time issue? If it is a run-time
>> issue,
>> would something like the procedure described below help address the
>> issue I'm
>> having? (I have taken this from the ATLAS documentation):
>>
>> "ATLAS does not provide a full lapack library. However, there is a
>> simple way
>> to get ATLAS to provide its faster LAPACK routines to a full LAPACK
>> library.
>> ATLAS's internal routines are distinct from LAPACK's, so it is safe to
>> compile
>> ATLAS's LAPACK routines directly into a netlib-style LAPACK library.
>> If you install LAPACK before ATLAS, this can be automated as described in
>> ATLAS/INSTALL.txt. The following describes adding netlib LAPACK to an
>> already built ATLAS library:
>>
>> First, obtain the LAPACK src from netlib and build the LAPACK library as
>> normal. Then, in the lib/ subdirectory of your ATLAS build directory
>> (where you should have a liblapack.a), issue the following commands:
>> mkdir tmp
>> cd tmp
>> ar x ../liblapack.a
>> cp <your LAPACK path & lib> ../liblapack.a
>> ar d ../liblapack.a lsame.o xerbla.o csrot.o zdrot.o
>> ar r ../liblapack.a *.o
>> cd ..
>> rm -rf tmp
>>
>> Just linking in ATLAS's liblapack.a first will not get you the best LAPACK
>> performance, mainly because LAPACK's untuned ILAENV will be used instead
>> of ATLAS's slightly tuned one."
>>
>> Thanks again,
>> Greg
>>
>> Quoting Gregory Magoon <gmagoon at mit.edu>:
>>
>>> An update: although the original command you suggested didn't work, as
>>> explained
>>> in my earlier e-mail, I tried the same command with the -noblas and
>>> -nolapack
>>> options that you proposed and the tests seem to be working properly:
>>>
>>> ./configure -batch -gcc -gfortran -mpp -auto-ga-tcgmsg-mpich2 -instroot
>>> /usr/local/molpro2010.1 -noblas -nolapack
>>>
>>> So, it seems our ATLAS, BLAS and/or LAPACK libraries were causing the
>>> issue.
>>> However, it is still not clear why. I had just installed these
>>> libraries as
>>> Ubuntu packages:
>>> sudo apt-get install libatlas-base-dev
>>> sudo apt-get install liblapack-dev
>>> Perhaps there is some critical step I omitted or these are not the
>>> correct
>>> packages?
>>>
>>> Thanks again,
>>> Greg
>>>
>>> user at kamet:~/Molpro$ ./configure -batch -gcc -gfortran -mpp
>>> -auto-ga-tcgmsg-mpich2 -instroot /usr/local/molpro2010.1 -noblas
>>> -nolapack
>>>
>>> machine type recognized as x86_64 (Generic 64-bit)
>>> kernel recognized as Linux
>>>
>>> user request compiler gfortran
>>> GNU Fortran Compiler, Version 4.5.2
>>> FC=/usr/bin/gfortran
>>>
>>> user request compiler gcc
>>> GNU Compiler Collection, Version 4.5.2
>>> CC=/usr/bin/gcc
>>>
>>> starting auto-build of prerequisites
>>>
>>> building MPICH2 version 1.3.3rc1, each step could take a few minutes
>>>
>>> ./configure --prefix=/home/user/Molpro/src/mpich2-install
>>> --with-device=ch3:nemesis --with-pm=hydra --enable-f77
>>> F77=/usr/bin/gfortran
>>> --enable-fc FC=/usr/bin/gfortran --enable-cc CC=/usr/bin/gcc --enable-cxx
>>> CXX=/usr/bin/g++
>>> make
>>> make install
>>> MPICH2 built, if you want to test the MPICH2 build, please run:
>>>
>>> cd /home/user/Molpro/src/mpich2-1.3.3rc1; make testing
>>>
>>> building Global Arrays version 5-0-2, each step could take a few minutes
>>>
>>> ./configure --prefix=/home/user/Molpro/src/ga-install --with-scalapack=no
>>> --enable-f77 F77=/usr/bin/gfortran CC=/usr/bin/gcc CXX=/usr/bin/g++
>>> --with-tcgmsg --with-mpi='/home/user/Molpro/src/mpich2-install/lib
>>> -lmpich
>>> -lopa -lmpl -lrt -I/home/user/Molpro/src/mpich2-install/include'
>>> make
>>> make install
>>> Global Arrays built, if you want to test the Global Arrays build,
>>> please run:
>>>
>>> cd /home/user/Molpro/src/ga-5-0-2; make checkprogs; make check
>>> MPIEXEC="/home/user/Molpro/src/mpich2-install/bin/mpiexec -np 4"
>>>
>>> ./configure -batch "-gcc" "-gfortran" "-mpp" "-instroot"
>>> "/usr/local/molpro2010.1" "-noblas" "-nolapack" -mppbase
>>> /home/user/Molpro/src/ga-5-0-2
>>>
>>> machine type recognized as x86_64 (Generic 64-bit)
>>> kernel recognized as Linux
>>>
>>> user request compiler gfortran
>>> GNU Fortran Compiler, Version 4.5.2
>>> FC=/usr/bin/gfortran
>>>
>>> user request compiler gcc
>>> GNU Compiler Collection, Version 4.5.2
>>> CC=/usr/bin/gcc
>>>
>>> ga_GA_MP_LIBS = -lmpich -lopa -lmpl -lrt
>>> ga_GA_MP_LDFLAGS = -L/home/user/Molpro/src/mpich2-install/lib
>>> ga_GA_MP_CPPFLAGS = -I/home/user/Molpro/src/mpich2-install/include
>>> ga_TARGET = LINUX64
>>> ga_MSG_COMMS = TCGMSGMPI
>>> ga_prefix = /home/user/Molpro/src/ga-install
>>> ga_ARMCI_NETWORK_LDFLAGS =
>>> ga_ARMCI_NETWORK_LIBS =
>>> ga_BLAS_SIZE = 4
>>>
>>> Use MPP library - Global Arrays version 5 or higher
>>> MPPLIB=-L/home/user/Molpro/src/ga-install/lib -lga -larmci
>>>
>>> MPILIB=-L/home/user/Molpro/src/mpich2-install/lib -lmpich -lopa -lmpl
>>> -lrt
>>>
>>> parallel=mpich2
>>>
>>> parse-Linux-x86_64-i8.o is your object
>>>
>>> CONFIG file created; proceed to compilation
>>>
>>> Quoting Andy May <MayAJ1 at cardiff.ac.uk>:
>>>
>>>> Greg,
>>>>
>>>> I can't seem to replicate this problem using options as close as
>>>> possible to yours.
>>>>
>>>> One thing I have noticed is that linking only the routines that come
>>>> with ATLAS does not work anymore with Molpro. I have fixed this, and
>>>> there will be an update sometime in the future, but this leads me to
>>>> conclude the -lapack you are using is either a supplemented ATLAS
>>>> library, or a system one.
>>>>
>>>> I don't think the BLAS/LAPACK will be the problem, but perhaps
>>>> configure has not correctly determined the integer size in these
>>>> libraries, you can always try using Molpro internal ones with -noblas
>>>> -nolapack options.
>>>>
>>>> There may be problems with the MPICH2 version which ships with
>>>> Ubuntu, certainly there were problems with hydra around 1.3.1, but if
>>>> you say that ifort works then maybe it's okay. To eliminate this I
>>>> would suggest to use a newer version (no need to do this by hand, see
>>>> below).
>>>>
>>>> Also, we are now at PL21 (there is a new tarball).
>>>>
>>>> Can I therefore suggest you first try the following with 2010.1.21:
>>>>
>>>> ./configure -batch -gcc -gfortran -mpp -auto-ga-tcgmsg-mpich2
>>>> -instroot /usr/local/molpro2010.1
>>>>
>>>> and see if this then gives you a working build.
>>>>
>>>> Best wishes,
>>>>
>>>> Andy
>>>>
>>>
>>>
>>>
>>>
>>
>>
>