[molpro-user] Benchmark timing on linux raid

[Jacek Klos]

Dear Molpro Users,

I have noticed some strange behavior on our linux Dell machine.  
Machine is dual-6core 3.33GHz
24GB RAM, and RAID disk composed of 3 SAS 15k RPM drives making  
logical volume of about 1TB.

When I run   small_normal_ccsd benchmarks with 12 cpus using parallel  
molpro (GA4.2 Open-MPI) and I repeat
it let's say 20 times after the fresh reboot of the machine the  
fastest elapsed time is 76 seconds.
The range of elapsed times is more or less +/- 5 sek.

But when I run again the same series of 12cpus jobs after machine is   
on for a day or so I get elapsed timings
like 30 seconds longer:

This is after the fresh reboot:

Timings for benchmark job small_normal_ccsd
  RUN ON Linux-2.6.18-194.26.1.el5/(x86_64) 64 bit
mpp version AT 21-Nov-10 11:49:04 WITH 12 PROCESSORS
  OUTPUT FILE: /opt/molpro_ga42_openmpi/molpro2010.1/bench/
small_normal_ccsd.n12_18.xml

  PROGRAM          ENERGY      USER    SYS  TOTCPU  ELAPSED  USER(%)
TOTCPU(%)
  AOINT          0.00000000    0.70   0.12    0.82     0.83
84.34      98.80
  AOSORT         0.00000000    2.61   0.79    3.40     3.55
73.52      95.77
  INT(TOT)       0.00000000    3.31   0.91    4.22     4.38
75.57      96.35
  HF-SCF      -156.15930716    1.17   0.13    1.30     1.31
89.31      99.24
  TRANSFORM      0.00000000    8.86   1.21   10.07    10.13
87.46      99.41
  CCSD        -156.83775164   21.17   1.34   22.51    22.84
92.69      98.56
  TRIPLES       -0.02781518   36.82   1.01   37.83    37.83
97.33     100.00
  CCSD(T)     -156.86556682   66.88   3.63   70.51    70.91
94.32      99.44
  TOTAL          0.00000000   71.36   4.67   76.03    76.60
93.16      99.26

This is the same job after machine ran for a day:

  PROGRAM          ENERGY      USER     SYS   TOTCPU  ELAPSED
USER(%)  TOTCPU(%)
  AOINT          0.00000000    1.14    0.51     1.65     1.72
66.28      95.93
  AOSORT         0.00000000    6.26    5.01    11.27    11.53
54.29      97.75
  INT(TOT)       0.00000000    7.40    5.52    12.92    13.25
55.85      97.51 HF-SCF      -156.15930716    2.26    0.70
2.96     2.98    75.84      99.33
  TRANSFORM      0.00000000   11.34    4.43    15.77    16.58
68.40      95.11
  CCSD        -156.83775164   21.81    4.81    26.62    29.90
72.94      89.03
  TRIPLES       -0.02781518   39.77    1.32    41.09    41.10
96.76      99.98
  CCSD(T)     -156.86556682   73.04   12.00    85.04    89.14
81.94      95.40
  TOTAL          0.00000000   82.70   18.22   100.92   105.37
78.49      95.78


  Seems like times for AOSORT, TRANSFORM are longer after machine is on
for some time.
Is this some RAID disk cache filling up and staying full?
Did anybody see such behavior before? Is it some drive cache filling up?

Best
Jacek Klos