Ryfujk

I am facing a performance issue with my small virtualization environnment.

I have two Dell PowerEdge 2950 and a Qnap TS-453-BU.
The Dells are equipped with XenServer 6.5.0 and they are attached to an iSCSI target with a pair of LUNs on the Qnap.

So far I installed several VM with no issues.
Recently I installed a few debian 8 with a LVM partitioning and i faced some performance issues.

To analyze them, i installed two twin vms, with debian 8, one with LVM (testlvm) and one without (testnolvm) and i ran some bonnie++ tests:

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

testnolvm        4G    89  99 47597  25 29559  26   127  99 95660  41 443.2  23

Latency               190ms     520ms     575ms     127ms     267ms    1584ms

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

testlvm          4G    85  99 48245  27 30707  27   120  99 94637  45 491.7  22

Latency               214ms     203ms     589ms     241ms     189ms     411ms



Version  1.97       ------Sequential Create------ --------Random Create--------

testnolvm           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16 12529  77 +++++ +++ 20241  89 16491  90 +++++ +++ 20389  90

Latency              2099us     637us     777us    1013us     270us     522us

Version  1.97       ------Sequential Create------ --------Random Create--------

testlvm             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16 14261  86 +++++ +++ 19722  90 16002  91 +++++ +++ 12159  56

Latency             29901us     580us     718us     232us    1236us     602us

I see very similar results on the throughputs but different in the latency.
I am not able to figure out what is wrong, because i also made other tests (mixing debian 7, debian 8, lvm, no lvm scenarios) getting confusing results.

It is a fact that i have a debian 7 with no lvm (installed at least one year ago) and a recently installed debian 8 with lvm that have very different performance;

for example, timing a "cat -file-" (i tried /etc/issue but also /proc/diskstats, so both real fs and memory) gives me a real of 0.005s on the first machine and 10 times more on the second machine.

Another example is that activating (on the second machine) a zabbix agent that monitors disk performance (making a lot of cat and grep on the /proc/diskstats) brings the cpu util to 50% without any other load (while on the old deb7 vm is almost seamless).

I even tried to move this "slow" vm on a new (thick provisioned) LUN without any improvement.

Any hints?
Thanks in advance.