[molpro-user] ARMCI test errors after compiling with GA/MPICH2, gfortran, ATLAS on Ubuntu 11.04

[Andy May]

Greg,

I think you need to explicitly tell ATLAS to compile with 8-byte 
integers with the appropriate compiler flags. Here is the script I used

# build ATLAS and install in /opt/atlas/3.8.4
mkdir -p /tmp/ATLAS/build
tar -C /tmp/ATLAS -xjf ~/atlas3.8.4.tar.bz2
cd /tmp/ATLAS/build
/tmp/ATLAS/ATLAS/configure --prefix=/opt/atlas/3.8.4 -Si cputhrchk 0 -C 
if `which gfortran` -F if '-O3 -m64 -fdefault-integer-8'
sed -i -e 's/F77_INTEGER=long/F77_INTEGER="long long"/' Make.inc
make
sudo make install

There is no need to cp/ln the libraries. You can tell GA where they are 
with --with-blas8='-L/opt/atlas/3.8.4/lib ...' etc., and you can point 
Molpro to the install with:

-blaspath /opt/atlas/3.8.4/lib

and it will do the rest.

Best wishes,

Andy

On 04/06/11 17:37, Gregory Magoon wrote:
> Thanks Andy,
> Unfortunately, when I tried approach #1, it gave the same run-time failure
> pattern in the tests and it was actually slower (at least for the
> big_lattice
> results I have been using as benchmark). I have gone back to using the
> regular
> blas and lapack ubuntu packages, which seems to work without error,
> though it
> seems to be a factor of 2+ slower.
>
> I've placed my installation notes for the failed attempt with 8-byte
> integer
> atlas below in case you can spot that I'm doing anything wrong.
> Greg
>
>
> #get dependencies
>
> sudo apt-get install mpich2
> sudo apt-get install curl
> sudo apt-get install gfortran
> sudo apt-get install zlib1g-dev
> sudo apt-get install libcr-dev
>
> #disable CPU throttling (http://ubuntuforums.org/showthread.php?t=611084)
> sudo /usr/bin/cpufreq-selector -g performance
>
> #install LAPACK (cf section 8 of ATLAS install guide:
> http://math-atlas.sourceforge.net/atlas_install/atlas_install.html#SECTION00090000000000000000
>
> )
> (in lapack directory)
> cp /INSTALL/make.gfortran.inc make.inc
> #edit make.inc to have the following:
> FORTRAN = gfortran
> OPTS = -O -fPIC -m64
> DRVOPTS = $(OPTS)
> NOOPT = -O0 -fPIC -m64
> LOADER = gfortran
> LOADOPTS = $(OPTS)
> TIMER = INT_ETIME
>
> make lib
>
>
> #from atlas build directory:
> ../configure -b 64 -A 0 -D c -DWALL -Fa alg -fPIC -Si archdef 0
> --incdir=/usr/include/ATLAS3.8.4 --libdir=/usr/lib64/ATLAS3.8.4
> --with-netlib-lapack=/home/user/lapack-3.3.1/lapack_LINUX.a
> make
> cd (build)/lib; make shared; make ptshared
> make check; make ptcheck
> make time
> sudo make install
> #manually copy dynamic libraries
> sudo cp lib/*.so /usr/lib64/ATLAS3.8.4/
> sudo chmod 0644 /usr/lib64/ATLAS3.8.4/*.so
>
> (remove existing lapack, blas library packages not needed: sudo apt-get
> remove)
>
> #point to the atlas libraries
> cd /usr/lib64
> sudo ln -s /usr/lib64/ATLAS3.8.4/liblapack.a liblapack.a
> sudo ln -s /usr/lib64/ATLAS3.8.4/liblapack.so liblapack.so
> sudo ln -s /usr/lib64/ATLAS3.8.4/libcblas.a libcblas.a
> sudo ln -s /usr/lib64/ATLAS3.8.4/libcblas.so libcblas.so
> sudo ln -s /usr/lib64/ATLAS3.8.4/libf77blas.a libf77blas.a
> sudo ln -s /usr/lib64/ATLAS3.8.4/libf77blas.so libf77blas.so
> sudo ln -s /usr/lib64/ATLAS3.8.4/libatlas.a libatlas.a
> sudo ln -s /usr/lib64/ATLAS3.8.4/libatlas.so libatlas.so
> sudo ln -s /usr/lib64/ATLAS3.8.4/libptcblas.a libptcblas.a
> sudo ln -s /usr/lib64/ATLAS3.8.4/libptcblas.so libptcblas.so
> sudo ln -s /usr/lib64/ATLAS3.8.4/libptf77blas.a libptf77blas.a
> sudo ln -s /usr/lib64/ATLAS3.8.4/libptf77blas.so libptf77blas.so
>
> #compile Global arrays **with LAPACK and 8-sized BLAS libraries**
> ./configure TARGET=LINUX64 FC=gfortran --with-mpi='-I/usr/include/mpich2
> -L/usr/lib64 -lmpichf90 -lmpich -lpthread -lmpl' --with-blas8='-L/usr/lib64
> -lf77blas -lcblas -latlas' --with-lapack='-L/usr/lib64 -llapack'
> make
> make check
> mpirun ./global/testing/test.x
>
> #(as root)
> sudo make install
> #will be installed in /usr/local/bin and /usr/local/lib
> copy config.log to /usr/local/config.log
>
> #compile molpro
> ./configure -batch -gcc -gfortran -mpp -mppbase /usr/local -instroot
> /usr/local/molpro2010.1
> make
>
> #tuning, testing
> make patch #enter username, password to download token
> make tuning
> nohup make MOLPRO_OPTIONS=-n8 test > test052511e.txt &
> #***original failure pattern, very slow; timing for big_lattice: 436/455
>
> Quoting Andy May <MayAJ1 at cardiff.ac.uk>:
>
>> Greg,
>>
>> Apologies for the delay, I was away last week.
>>
>> The problem with Molpro/ATLAS was a compile time problem. Essentially
>> Molpro should be able to be built with just the ATLAS routines, and
>> should automatically build the missing LAPACK routines, thus avoiding
>> the user to do this manually. This is now fixed in our repository, and
>> will be pushed out with some other fixes after testing. (Some LAPACK
>> routines not present in ATLAS were also not being compiled by Molpro).
>>
>> I suspect the problem is that for ATLAS we assume it has been built
>> with 8-byte integers on 64-bit systems, and probably the ubuntu
>> binaries are built with 4-byte integers.
>>
>> Also Molpro configure has a dilemma when it finds libatlas and then
>> liblapack because the name 'liblapack' is rather generic. We assume
>> that liblapack is probably an ATLAS LAPACK library since libatlas is
>> present, but it is perfectly possible liblapack could be a system one
>> (on openSUSE for example liblapack is simply the netlib routines). So
>> the fact your liblapack is a full LAPACK library has been ignored, and
>> Molpro will have compiled the non-ATLAS routines internally (with
>> 8-byte integers), which will add to the confusion.
>>
>> To use ATLAS you could either:
>>
>> 1. Compile ATLAS by hand with 8-byte integers (this will also give you
>> optimal parameters for your hardware). Then build GA again using
>> --with-blas8 option, and then finally build Molpro. This will then
>> give you everything with 8-byte integers.
>>
>> 2. Proceed as before, but after configure edit CONFIG and change the
>> values of BLAS_INT and LAPACK_INT in the MACROS variable. I think this
>> should work, but haven't tried it.
>>
>> Best wishes,
>>
>> Andy
>>
>> On 23/05/11 18:23, Gregory Magoon wrote:
>>> I was able to narrow this down to an issue with ATLAS. Andy, could you
>>> elaborate
>>> on your previous comments regarding Molpro not working with the
>>> unmodified ATLAS
>>> libraries? Is this a compile-time or run-time issue? If it is a run-time
>>> issue,
>>> would something like the procedure described below help address the
>>> issue I'm
>>> having? (I have taken this from the ATLAS documentation):
>>>
>>> "ATLAS does not provide a full lapack library. However, there is a
>>> simple way
>>> to get ATLAS to provide its faster LAPACK routines to a full LAPACK
>>> library.
>>> ATLAS's internal routines are distinct from LAPACK's, so it is safe to
>>> compile
>>> ATLAS's LAPACK routines directly into a netlib-style LAPACK library.
>>> If you install LAPACK before ATLAS, this can be automated as
>>> described in
>>> ATLAS/INSTALL.txt. The following describes adding netlib LAPACK to an
>>> already built ATLAS library:
>>>
>>> First, obtain the LAPACK src from netlib and build the LAPACK library as
>>> normal. Then, in the lib/ subdirectory of your ATLAS build directory
>>> (where you should have a liblapack.a), issue the following commands:
>>> mkdir tmp
>>> cd tmp
>>> ar x ../liblapack.a
>>> cp <your LAPACK path & lib> ../liblapack.a
>>> ar d ../liblapack.a lsame.o xerbla.o csrot.o zdrot.o
>>> ar r ../liblapack.a *.o
>>> cd ..
>>> rm -rf tmp
>>>
>>> Just linking in ATLAS's liblapack.a first will not get you the best
>>> LAPACK
>>> performance, mainly because LAPACK's untuned ILAENV will be used instead
>>> of ATLAS's slightly tuned one."
>>>
>>> Thanks again,
>>> Greg
>>>
>>> Quoting Gregory Magoon <gmagoon at mit.edu>:
>>>
>>>> An update: although the original command you suggested didn't work, as
>>>> explained
>>>> in my earlier e-mail, I tried the same command with the -noblas and
>>>> -nolapack
>>>> options that you proposed and the tests seem to be working properly:
>>>>
>>>> ./configure -batch -gcc -gfortran -mpp -auto-ga-tcgmsg-mpich2 -instroot
>>>> /usr/local/molpro2010.1 -noblas -nolapack
>>>>
>>>> So, it seems our ATLAS, BLAS and/or LAPACK libraries were causing the
>>>> issue.
>>>> However, it is still not clear why. I had just installed these
>>>> libraries as
>>>> Ubuntu packages:
>>>> sudo apt-get install libatlas-base-dev
>>>> sudo apt-get install liblapack-dev
>>>> Perhaps there is some critical step I omitted or these are not the
>>>> correct
>>>> packages?
>>>>
>>>> Thanks again,
>>>> Greg
>>>>
>>>> user at kamet:~/Molpro$ ./configure -batch -gcc -gfortran -mpp
>>>> -auto-ga-tcgmsg-mpich2 -instroot /usr/local/molpro2010.1 -noblas
>>>> -nolapack
>>>>
>>>> machine type recognized as x86_64 (Generic 64-bit)
>>>> kernel recognized as Linux
>>>>
>>>> user request compiler gfortran
>>>> GNU Fortran Compiler, Version 4.5.2
>>>> FC=/usr/bin/gfortran
>>>>
>>>> user request compiler gcc
>>>> GNU Compiler Collection, Version 4.5.2
>>>> CC=/usr/bin/gcc
>>>>
>>>> starting auto-build of prerequisites
>>>>
>>>> building MPICH2 version 1.3.3rc1, each step could take a few minutes
>>>>
>>>> ./configure --prefix=/home/user/Molpro/src/mpich2-install
>>>> --with-device=ch3:nemesis --with-pm=hydra --enable-f77
>>>> F77=/usr/bin/gfortran
>>>> --enable-fc FC=/usr/bin/gfortran --enable-cc CC=/usr/bin/gcc
>>>> --enable-cxx
>>>> CXX=/usr/bin/g++
>>>> make
>>>> make install
>>>> MPICH2 built, if you want to test the MPICH2 build, please run:
>>>>
>>>> cd /home/user/Molpro/src/mpich2-1.3.3rc1; make testing
>>>>
>>>> building Global Arrays version 5-0-2, each step could take a few
>>>> minutes
>>>>
>>>> ./configure --prefix=/home/user/Molpro/src/ga-install
>>>> --with-scalapack=no
>>>> --enable-f77 F77=/usr/bin/gfortran CC=/usr/bin/gcc CXX=/usr/bin/g++
>>>> --with-tcgmsg --with-mpi='/home/user/Molpro/src/mpich2-install/lib
>>>> -lmpich
>>>> -lopa -lmpl -lrt -I/home/user/Molpro/src/mpich2-install/include'
>>>> make
>>>> make install
>>>> Global Arrays built, if you want to test the Global Arrays build,
>>>> please run:
>>>>
>>>> cd /home/user/Molpro/src/ga-5-0-2; make checkprogs; make check
>>>> MPIEXEC="/home/user/Molpro/src/mpich2-install/bin/mpiexec -np 4"
>>>>
>>>> ./configure -batch "-gcc" "-gfortran" "-mpp" "-instroot"
>>>> "/usr/local/molpro2010.1" "-noblas" "-nolapack" -mppbase
>>>> /home/user/Molpro/src/ga-5-0-2
>>>>
>>>> machine type recognized as x86_64 (Generic 64-bit)
>>>> kernel recognized as Linux
>>>>
>>>> user request compiler gfortran
>>>> GNU Fortran Compiler, Version 4.5.2
>>>> FC=/usr/bin/gfortran
>>>>
>>>> user request compiler gcc
>>>> GNU Compiler Collection, Version 4.5.2
>>>> CC=/usr/bin/gcc
>>>>
>>>> ga_GA_MP_LIBS = -lmpich -lopa -lmpl -lrt
>>>> ga_GA_MP_LDFLAGS = -L/home/user/Molpro/src/mpich2-install/lib
>>>> ga_GA_MP_CPPFLAGS = -I/home/user/Molpro/src/mpich2-install/include
>>>> ga_TARGET = LINUX64
>>>> ga_MSG_COMMS = TCGMSGMPI
>>>> ga_prefix = /home/user/Molpro/src/ga-install
>>>> ga_ARMCI_NETWORK_LDFLAGS =
>>>> ga_ARMCI_NETWORK_LIBS =
>>>> ga_BLAS_SIZE = 4
>>>>
>>>> Use MPP library - Global Arrays version 5 or higher
>>>> MPPLIB=-L/home/user/Molpro/src/ga-install/lib -lga -larmci
>>>>
>>>> MPILIB=-L/home/user/Molpro/src/mpich2-install/lib -lmpich -lopa -lmpl
>>>> -lrt
>>>>
>>>> parallel=mpich2
>>>>
>>>> parse-Linux-x86_64-i8.o is your object
>>>>
>>>> CONFIG file created; proceed to compilation
>>>>
>>>> Quoting Andy May <MayAJ1 at cardiff.ac.uk>:
>>>>
>>>>> Greg,
>>>>>
>>>>> I can't seem to replicate this problem using options as close as
>>>>> possible to yours.
>>>>>
>>>>> One thing I have noticed is that linking only the routines that come
>>>>> with ATLAS does not work anymore with Molpro. I have fixed this, and
>>>>> there will be an update sometime in the future, but this leads me to
>>>>> conclude the -lapack you are using is either a supplemented ATLAS
>>>>> library, or a system one.
>>>>>
>>>>> I don't think the BLAS/LAPACK will be the problem, but perhaps
>>>>> configure has not correctly determined the integer size in these
>>>>> libraries, you can always try using Molpro internal ones with -noblas
>>>>> -nolapack options.
>>>>>
>>>>> There may be problems with the MPICH2 version which ships with
>>>>> Ubuntu, certainly there were problems with hydra around 1.3.1, but if
>>>>> you say that ifort works then maybe it's okay. To eliminate this I
>>>>> would suggest to use a newer version (no need to do this by hand, see
>>>>> below).
>>>>>
>>>>> Also, we are now at PL21 (there is a new tarball).
>>>>>
>>>>> Can I therefore suggest you first try the following with 2010.1.21:
>>>>>
>>>>> ./configure -batch -gcc -gfortran -mpp -auto-ga-tcgmsg-mpich2
>>>>> -instroot /usr/local/molpro2010.1
>>>>>
>>>>> and see if this then gives you a working build.
>>>>>
>>>>> Best wishes,
>>>>>
>>>>> Andy
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>