S4 app servers performance comparison: Redhat, Suse or Win2012, who's the winner?

So now our envrionment is kind of stable, and it's time to kick off various of performance workbench "race course" under certain conditions.

Drag race?!

First all, we would like to see how the batch jobs processing time goes. Again, I'll have to post our famous 'mix OS' architecture

Now we randomly pick a standard system batch job. The only reason we choose this specific job is because it runs a little longer, but not too long, like 1~2 minutes. We don't want to choose those jobs running few seconds, eitherway we have a hard time to choose a long running jobs like 30 minutes+, which we have not had that much data to run.

Of course we can load a billion line of data into our backend HANA DB, however, at this point, it's not necessary. Making a batch job running longer we have various of ways, for example, load some kind of programs to calculate the password against dictionary. I bet this type of 'ultimate calculation' program will run for days. Loading huge data int,o backend HANA DB will also impact the database buffer, app server buffer, swap and so on, which will make things complicated.

So we'll start testing the performance, from a very simple job, standard SAP job which is about to calculcate the UI5 index. Below is the job step definition

Now we create 3 batch jobs. Each job has exactly the same ABAP step as above. In order to make the job running in the specific application server, we'll have to fill in the 'Target' app server field when the job runs.

Below you can see how each of these jobs are defined. Then we release them to be run at the same time. Here we would also like to see if there are any priorities when these job runs, even though we did not define anything specific, such as the job class

Jobs are all released

OK. All jobs should be finished by now. Let's see the first 'race' result:

What we can see here is:

• The Redhat Linux app server ran the job first, which got 9 seconds delay, the during time was 110 seconds
• The Win2012 app server ran the job secondly, which had 18 seconds delay, and the during time was 115 seconds.
• The Suse Linux app server ran the last, which had 36 seconds delay, and ran for 104 seconds.

In above testing, we can see the winner was Suse, and the Loser was Win2012

In our current settings, all these 3 app servers have exactly same hardware configurations. To be able to confirm this, we double-checked the VMWare ESXi Guest VM configuration: they all have 2v CPU, 8GB ram, and shared same RAID5 disks

Well, if looking above 3 VMs settings, we can see the only difference was the SUSE linux did not have VMware Tools installed, would that be a reason?

To be able to get rid of any unforseen items, we decided to install the VMWare tools on both SUSE and Redhat Linux, then redo the test

S4APP2:/tmp/vmtools/vmware-tools-distrib # ls -ltr
total 500
drwxr-xr-x  3 root root   4096 Jan 29  2015 vgauth
drwxr-xr-x 14 root root   4096 Jan 29  2015 lib
-rwxr-xr-x  1 root root 199096 Jan 29  2015 vmware-install.pl
drwxr-xr-x  2 root root   4096 Jan 29  2015 installer
drwxr-xr-x  5 root root   4096 Jan 29  2015 etc
drwxr-xr-x  2 root root   4096 Jan 29  2015 doc
drwxr-xr-x  2 root root   4096 Jan 29  2015 bin
-rw-r--r--  1 root root   2538 Jan 29  2015 INSTALL
-rw-r--r--  1 root root 272678 Jan 29  2015 FILES
S4APP2:/tmp/vmtools/vmware-tools-distrib # ./vmware-install.pl
Creating a new VMware Tools installer database using the tar4 format.

Installing VMware Tools.

In which directory do you want to install the binary files?
[/usr/bin]

What is the directory that contains the init directories (rc0.d/ to rc6.d/)?
[/etc/init.d]

What is the directory that contains the init scripts?
[/etc/init.d]

In which directory do you want to install the daemon files?
[/usr/sbin]

In which directory do you want to install the library files?
[/usr/lib/vmware-tools]

The path "/usr/lib/vmware-tools" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
[yes]

In which directory do you want to install the documentation files?
[/usr/share/doc/vmware-tools]

The path "/usr/share/doc/vmware-tools" does not exist currently. This program
is going to create it, including needed parent directories. Is this what you
want? [yes]

The installation of VMware Tools 9.10.0 build-2476743 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command: "/usr/bin/vmware-uninstall-tools.pl".

Before running VMware Tools for the first time, you need to configure it by
invoking the following command: "/usr/bin/vmware-config-tools.pl". Do you want
this program to invoke the command for you now? [yes]

Initializing...


Making sure services for VMware Tools are stopped.

Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                                      done
   VMware User Agent (vmware-user):                                    done
   Unmounting HGFS shares:                                             done
   Guest filesystem driver:                                            done



Before you can compile modules, you need to have the following installed...

make
gcc
kernel headers of the running kernel


Searching for GCC...
Detected GCC binary at "/usr/bin/gcc".
The path "/usr/bin/gcc" appears to be a valid path to the gcc binary.
Would you like to change it? [no]

Searching for a valid kernel header path...
Detected the kernel headers at "/lib/modules/3.0.76-0.7-default/build/include".

The path "/lib/modules/3.0.76-0.7-default/build/include" appears to be a valid
path to the 3.0.76-0.7-default kernel headers.
Would you like to change it? [no]

Using kernel build system.
make: Entering directory `/tmp/modconfig-2kIYdY/vmci-only'
/usr/bin/make -C /lib/modules/3.0.76-0.7-default/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C ../../../linux-3.0.76-0.7 O=/usr/src/linux-3.0.76-0.7-obj/x86_64/default/. modules
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/linux/driver.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/linux/vmciKernelIf.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciContext.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciDatagram.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciDoorbell.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciDriver.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciEvent.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciHashtable.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciQPair.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciQueuePair.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciResource.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/common/vmciRoute.o
  CC [M]  /tmp/modconfig-2kIYdY/vmci-only/driverLog.o
  LD [M]  /tmp/modconfig-2kIYdY/vmci-only/vmci.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/modconfig-2kIYdY/vmci-only/vmci.mod.o
  LD [M]  /tmp/modconfig-2kIYdY/vmci-only/vmci.ko
make[1]: Leaving directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C $PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= postbuild
make[1]: Entering directory `/tmp/modconfig-2kIYdY/vmci-only'
make[1]: `postbuild' is up to date.
make[1]: Leaving directory `/tmp/modconfig-2kIYdY/vmci-only'
cp -f vmci.ko ./../vmci.o
make: Leaving directory `/tmp/modconfig-2kIYdY/vmci-only'

Using kernel build system.
make: Entering directory `/tmp/modconfig-kgMbgB/vmci-only'
/usr/bin/make -C /lib/modules/3.0.76-0.7-default/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C ../../../linux-3.0.76-0.7 O=/usr/src/linux-3.0.76-0.7-obj/x86_64/default/. modules
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/linux/driver.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/linux/vmciKernelIf.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciContext.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciDatagram.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciDoorbell.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciDriver.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciEvent.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciHashtable.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciQPair.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciQueuePair.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciResource.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/common/vmciRoute.o
  CC [M]  /tmp/modconfig-kgMbgB/vmci-only/driverLog.o
  LD [M]  /tmp/modconfig-kgMbgB/vmci-only/vmci.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/modconfig-kgMbgB/vmci-only/vmci.mod.o
  LD [M]  /tmp/modconfig-kgMbgB/vmci-only/vmci.ko
make[1]: Leaving directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C $PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= postbuild
make[1]: Entering directory `/tmp/modconfig-kgMbgB/vmci-only'
make[1]: `postbuild' is up to date.
make[1]: Leaving directory `/tmp/modconfig-kgMbgB/vmci-only'
cp -f vmci.ko ./../vmci.o
make: Leaving directory `/tmp/modconfig-kgMbgB/vmci-only'
Using kernel build system.
make: Entering directory `/tmp/modconfig-kgMbgB/vsock-only'
/usr/bin/make -C /lib/modules/3.0.76-0.7-default/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C ../../../linux-3.0.76-0.7 O=/usr/src/linux-3.0.76-0.7-obj/x86_64/default/. modules
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/linux/af_vsock.o
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/linux/notify.o
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/linux/notifyQState.o
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/linux/stats.o
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/linux/util.o
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/linux/vsockAddr.o
  CC [M]  /tmp/modconfig-kgMbgB/vsock-only/driverLog.o
  LD [M]  /tmp/modconfig-kgMbgB/vsock-only/vsock.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/modconfig-kgMbgB/vsock-only/vsock.mod.o
  LD [M]  /tmp/modconfig-kgMbgB/vsock-only/vsock.ko
make[1]: Leaving directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C $PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= postbuild
make[1]: Entering directory `/tmp/modconfig-kgMbgB/vsock-only'
make[1]: `postbuild' is up to date.
make[1]: Leaving directory `/tmp/modconfig-kgMbgB/vsock-only'
cp -f vsock.ko ./../vsock.o
make: Leaving directory `/tmp/modconfig-kgMbgB/vsock-only'

The module vmxnet3 has already been installed on this system by another
installer or package and will not be modified by this installer.

The module pvscsi has already been installed on this system by another
installer or package and will not be modified by this installer.

The module vmmemctl has already been installed on this system by another
installer or package and will not be modified by this installer.

The VMware Host-Guest Filesystem allows for shared folders between the host OS
and the guest OS in a Fusion or Workstation virtual environment.  Do you wish
to enable this feature? [no]

Using kernel build system.
make: Entering directory `/tmp/modconfig-Uc4ntf/vmxnet-only'
/usr/bin/make -C /lib/modules/3.0.76-0.7-default/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C ../../../linux-3.0.76-0.7 O=/usr/src/linux-3.0.76-0.7-obj/x86_64/default/. modules
  CC [M]  /tmp/modconfig-Uc4ntf/vmxnet-only/vmxnet.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/modconfig-Uc4ntf/vmxnet-only/vmxnet.mod.o
  LD [M]  /tmp/modconfig-Uc4ntf/vmxnet-only/vmxnet.ko
make[1]: Leaving directory `/usr/src/linux-3.0.76-0.7-obj/x86_64/default'
/usr/bin/make -C $PWD SRCROOT=$PWD/. \
          MODULEBUILDDIR= postbuild
make[1]: Entering directory `/tmp/modconfig-Uc4ntf/vmxnet-only'
make[1]: `postbuild' is up to date.
make[1]: Leaving directory `/tmp/modconfig-Uc4ntf/vmxnet-only'
cp -f vmxnet.ko ./../vmxnet.o
make: Leaving directory `/tmp/modconfig-Uc4ntf/vmxnet-only'

The vmblock enables dragging or copying files between host and guest in a
Fusion or Workstation virtual environment.  Do you wish to enable this feature?
[no]

VMware automatic kernel modules enables automatic building and installation of
VMware kernel modules at boot that are not already present. This feature can
be enabled/disabled by re-running vmware-config-tools.pl.

Would you like to enable VMware automatic kernel modules?
[no]


Disabling timer-based audio scheduling in pulseaudio.

Do you want to enable Guest Authentication (vgauth)? [yes]



Detected X server version 1.6.5


Unable to detect guest resolution.


Please choose one of the following display sizes that X will start with:

[1]< 640x480
[2]  800x600
[3]  1024x768
[4]  1280x800
Please enter a number between 1 and 4:

[1]


X is running fine with the new config file.

Creating a new initrd boot image for the kernel.

Kernel image:   /boot/vmlinuz-3.0.76-0.7-default
Initrd image:   /boot/initrd-3.0.76-0.7-default
Root device:    /dev/sda2 (mounted on / as ext3)
Resume device:  /dev/sda1
Kernel Modules: hwmon thermal_sys thermal processor fan scsi_mod scsi_transport_spi mptbase mptscsih mptspi libata ata_piix ata_generic vmxnet3 vmw_pvscsi vmxnet scsi_dh scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_dh_hp_sw mbcache jbd ext3 crc-t10dif sd_mod
Features:       acpi block resume.userspace resume.kernel
Bootsplash:     SLES (800x600)
33942 blocks
>>> Network: auto
>>> Calling mkinitrd -k /boot/vmlinuz-3.0.76-0.7-default -i /tmp/mkdumprd.O3R7LYrIU6 -f 'kdump network' -B  -s ''
Regenerating kdump initrd ...

Kernel image:   /boot/vmlinuz-3.0.76-0.7-default
Initrd image:   /tmp/mkdumprd.O3R7LYrIU6
Root device:    /dev/sda2 (mounted on / as ext3)
Resume device:  /dev/sda1
Kernel Modules: hwmon thermal_sys thermal processor fan scsi_mod scsi_transport_spi mptbase mptscsih mptspi libata ata_piix ata_generic vmxnet3 vmw_pvscsi vmxnet scsi_dh scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_dh_hp_sw mbcache jbd ext3 af_packet nls_utf8 crc-t10dif sd_mod
Features:       acpi block network resume.userspace resume.kernel kdump
47435 blocks
   Checking acpi hot plug                                              done
Starting VMware Tools services in the virtual machine:
   Switching to guest configuration:                                   done
   VM communication interface:                                         done
   VM communication interface socket family:                           done
   Guest operating system daemon:                                      done
   VGAuthService:                                                      done
The configuration of VMware Tools 9.10.0 build-2476743 for Linux for this
running kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking "/usr/bin/vmware-toolbox-cmd" from the
command line.

To enable advanced X features (e.g., guest resolution fit, drag and drop, and
file and text copy/paste), you will need to do one (or more) of the following:
1. Manually start /usr/bin/vmware-user
2. Log out and log back into your desktop session; and,
3. Restart your X session.

Enjoy,

--the VMware team


Found VMware Tools CDROM mounted at /media/VMware Tools. Ejecting device
/dev/sr0 ...
S4APP2:/tmp/vmtools/vmware-tools-distrib #

Now the SUSE has the VMWare tools running as well

All right, now we redo the same test again, copy the same job and release them to run at the same time. To be able to fair and get rid of any random impact on the servers, we repeat the same testing 3 times

From the result, we can see:

• the Delay time gap remain as 9 seconds, which is always Redhat runs the first, then Win2012, then SUSE. What caused this? When we defined these jobs, we did not put any priorities
• It's always SUSE runs the fastest, second is Redhat, and Win2012 is the slowest

One thing could impact the performance, of course, is the SAP application server kernels. We double-checked the kernels and make sure they're in the same patch level, which indeed they are

This is end of the first 'race course', scenario One.

Conclusion:

• With same hardware configuration, definitely there are performance winner and loser. We have seen many hardware vendors declare their hardware is "xxx times faster than the other", "yyy times faster than older version", but these kind of measurement is useless. We have witnessed a lot of clients after they spent tons of $$$ and upgraded their hardware to latest, but they didn't really feel it's any faster than before. One of the reason was ,the OS that running the SAP did not changed, still remained as the existing ones.

• For SAP, we have seen many workbench data such as 'this platform / appliance we have xxxx SAPS', but again, without comparison, no matter how many SAPS it declared, it's not really practical (note: the definition of SAPS see http://global.sap.com/campaigns/benchmark/measuring.epx)

• OLTP or OLAP performance? We know most of the SAP systems especially ECC6 or the later S4, they're typical OLTP systems. Different than OLAP which is more relied on the database performance, the OLTP systems, which based on transactions interactions, somehow also depend on how the application servers tuning. In the recent renovation of hardware and technologies, we can see shrinking down the footprint of application servers would be necessary, regarding cost saving and ease of management. Talking about load balancing, it would be reduced from let say, each region has 100 servers, to be maybe 10 servers. However, to be able to choose the best or most suitable OS to run SAP, the application servers would be crucial.

• Next workbench testing? Yes we will have more scenarios to come in the next few weeks. We'll keep you posted!

Summary chart see below, the shorter, the better!