[PaulClayton]

Have a question on the above using VIO to offer up SCSI disks, versus going direct. What is the performance loss when using VIO versus direct attached.

My tests show up about 35% loss using VIO.
Any thought on this?

[ross.mather]

It all depends on your I/O pattern. If you use the VIO and its LVM then its a big hit. If you give each LPAR its own disks then you only have to arbitrate the SCSI bus.

What exactly are you testing?

[PaulClayton]

My testing is fairly simple.
SAN<-->SVC<-->VIO<-->LPAR

I pass the disk straight through, and don't use LVM on the VIO for this.

Simply testing on the client LPAR this command
dd if=/dev/zero of=/test/test count=102400 bs=8192
I get about 47Mbytes per sec.

Now remove the device from the LPAR, and create a LVM on the VIO, and run the same test on the same physical device, I get 70Mbytes per sec.
Hence the question, of is this normal. I can't find anything, that would suggest that this is, but I did find some docs, that suggested, that IO would be significantl;y reduced in the VIO configuration, and higher throughput would require directly attached disk to obtain maximum. I am trying to obtain some baseline numbers to ensure that my set up is correct, and we don't have an underlying problem anywhere.

[mdulson]

Hi,

I've had the same questions myself, and found the attached pdf lurking on the IBM website. My initial findings on using direct versus virtualised disks (when pasing through SAN based LUNS rather than LVM based) couldn't find any performance difference - it's worth bearing in mind that the effect of cache in SAN arrays means that repeating the same experiment twice can see vastly improved/differing results.

It's also worth bearing in mind that if the disks are being used in a cluster, that you're clustering software supports virtualised disks (I seem to recall that HACMP didn't support VIOS supplied disk that were being provided by an SVC


Matt

[PaulClayton]

I think there is a cut of point with virtualized disks in performance. We have some LVMs on our VIOS, that have 30 LVs on them, and performance really sucks. Max I can get out these disks is 14Mbyte per second versus 200Mbyte per per second on direct attached.
I think part of the problem lays with the initial set up of the LVM on VIOS, in that the disks were not configure in a stripe, and only a concat.
From a maintenance aspect, passing the disks through is far less complex, and if you run dual VIOS, then I would recomend going direct disk rather than virtual disks, as LVM adds another layer of complexity in that you have to set concurrent LVM on your VIOS to get a dual VIOS operation.

[ross.mather]

Paul, I've tried striping the disks in a RAID set and then using the LVM on that set - and the performance is just as bad as when you use concat Volume Groups. The troble is the Logical Volume Manager at the VIO level (in my opinion).

Every write you do at an LPAr level has to go through the chain:
OS Write to File System -> OS Write to Logical Volume -> OS Write to Virtual Disk -> VIO Wite to Logical Volume -> VIo Write to Physical Disks. There are simply too many layers in there for there to vbe a decent performance.

Its a handy thing to be able to do for a aplay around environemnt, but for any production environment they LPARS need eithe r pass through Physcal or SAN disks and not LVs off of shared disks.

incidentally - my timings and exams largely concur with yours.

[PaulClayton]

IBM seem to recommend this process, but I suppose for cash strapped customers, the LVM virtual disk route is OK, providing you know that
A) Performance may be an issue.
B) moving your LVMs to disk pass through is not going to possible unless you have addition disk to migrate.

One of the advantages we have found with this disk pass through, is the ease of migrating systems from our P5 to P6 environment.
On the P6 we set up disk pass through, configured the LVMs on the client partition, then exported the VG and attached them to the P5 VIO, and set them up as pass through on the P5 to the client partition. Re-imported, copied the data, and reversed the operation.
When doing the data copies, my read disk ran at 100% busy and my disks to be written to were running between 6-10% busy.

[Administrator]

Areyou making sure that you change the queue depth on the LPAR to match the queue depth on the VIO server?

The default settings are:

fibre channel VIO - Queue depth of 10
VSCSI on LPAR - Queue depth of 3

What you need to do is tune both values properly. The LUN at the VIO server should have a queue depth equal to 3 times the number of physical devices that support the LUN.

I have a 2 TB LUN, which I assign to my VIO servers. It is an SVC vdisk. The mdisk group it was provisioned from is made up of 8 7+P arrays from my DS8300.

That means my VIO LUN is created from 56 drives and should have a queue depth of 168.

On the VIO servers:
chdev -dev hdiskX -attr queue_depth=168 pv=yes reserve_policy=no_reserve

Now create the vdev

On the LPAR, after running cfgmgr but before anything else:
chdev -l hdiskX -a queue_depth=168

Rerun your tests after tuning queue_depth and see if there is any improvement.

[PaulClayton]

Interesting. I tried your suggestion, and it has made a difference. Previously the max read throughput I could get on a specific server was 14Mbyte. This has jumped to 57Mbyte per sec.