Raxco has been working with VMware the past several months to assess the impact of Windows guest defragmentation on VMware performance. The most recent testing used VMware's vscsiStats utility to capture metrics. The vscsiStats utility sits between the VMware kernel and the Monitor. It captures every IO request through the storage stack and sorts them into various buckets that can be used for performance analysis. We ran our test concurrently using five virtual machines on an ESX 4.0 cluster.
The goal of the test was to determine if Windows guest defragmentation improves virtualization performance. The test was fairly simple; we created a fragmented disk with ISO images and other files and made a copy for each of the five virtual machines. We installed MS Office and MS SQL on each of the VMDKs and measured the results with vscsiStats. After the data collection was done on the fragmented disks, we used PerfectDisk to defragment the files and free space on all of the disks and repeated the tests using the defragmented disks.
The results were pretty impressive on several fronts. First of all, one of the main sources of virtualization overhead occurs when IO have to traverse the virtual storage stack. Our testing showed that when the Windows guests were defragmented with PerfectDisk the total number of IO was reduced by 36.5% and write IO was reduced by 44.6% after the MS SQL installation. This means defragmentation was able to reduce the total IO load by over 1/3 for all IO and it reduced write IO by almost half. How was defragmentation able to do this?
The answer is that while file defragmentation improves file read time, free space consolidation improves file write time by reducing the number of physical accesses needed to write the file. If the guest file system can find sufficient contiguous free space, the likelihood of the file being written in one piece improves. Contiguous free space also means you do bigger writes. The vscsiStats bear this out. The PerfectDisk disk created 11.9 times as many IO greater than 524K, which is the largest vscsiStats bucket size. The vscsiStats histogram illustrates this difference on the far right column.
Disk IO latency was also affected by the file defragmentation and free space consolidation of the Windows guests. As noted above, the Windows file system was able to perform fewer and larger IO when the disk is defragmented and the free space is aggregated in a large chunk. It logically follows that these IO will complete in less time and the vscsiStats again support this assumption. The utility sorts IO into buckets ranging from .001ms to >100ms. Since the average disk drive access time is about 15ms, we viewed all IO taking more than 15ms as a slow IO. The vscsiStats show the defragmented disk reduced the total number of IO taking longer than 15ms by 48.7%.
The final metric that showed improvement due to guest system defragmentation was sequential IO. The vscsiStats utility measures the distance in logical blocks between seeks. This histogram shows that the defragmented disk increased the number of IO that were only one block away (center columns) and decreased the number of IO that were 1000 to 500,000 logical blocks away (left and rightmost columns).
In summary, it appears Windows guest file defragmentation and free space consolidation will:
- Reduce VMware overhead by reducing the total number of IO that traverse the virtual storage stack
- Improve system throughput by producing larger IO
- Reduce system latency by reducing total IO and generating larger IO
- Improve virtualization performance by increasing sequential IO
- Reduce the demand for host CPU, memory and IO resources
- Potentially create a situation where the host can support one or more additional guests due to the resources conserved
To access a complete copy of the white paper detailing the test procedures and results go to
www.perfectdisk.com/user_data/white_papers/vmware_multi_test_new.pdf
From time to time, I’ve saluted Microsoft MVPs (Most Valuable Professionals), recognizing the invaluable service they provide to the Microsoft user community in numerous categories. Having an MVP on staff here has only deepened my appreciation for what they do. Of course, I have to admit it’s nice to know and makes us proud that PerfectDisk is the preferred defragmenter of choice of MVPs around the world.
I love hearing their stories, which are relayed to us frequently. Keith Elder is a Microsoft MVP and INETA speaker, and runs the popular, informative and entertaining Deep Fried Bytes technology podcast as well as his blog, Words of Wisdom from the Elder. Keith is a big fan of the PerfectDisk base product, but also has experienced huge benefits from the PerfectDisk VMware Bundle.
Keith had a virtual machine that would not boot. He kept receiving an error message and it appeared to him that the disk was somewhat corrupted. Keith says he “ran PerfectDisk on it and after it was done it fixed the problem, and I’m still developing in this VM right now. Had it not been for PerfectDisk’s integration with VMware, I honestly don’t know what I would have done next. Probably would have had to resort to recovering off site backups.”
How do you defrag VMware and boost virtualization performance?
The PerfectDisk 11 Client and Enterprise Console have been updated to Build 174. These latest builds include various bug fixes and minor enhancements.
You can “Check For Updates” in the products or wait for Autoupdate.
Client enhancements and fixes:
- Correct issues connecting to and displaying data stores for Exchange 2010
- Change behavior when stopping a defrag pass. PerfectDisk used to display the statistics at the time you stopped even though they were inaccurate. PerfectDisk no longer does this.
- The display files in block feature has been disabled during a defrag and analyze pass since there was no data to display
- Corrected some display issues on the virtual machine tab
- Added support for translation into Japanese
- Corrected some statistical calculations when files are being added and deleted from the drive during the defragmentation pass.
- The list of drive types that can be impersonated was limited to prevent confusion. The purpose of impersonating drive types was due to the inability to accurately determine whether or not a drive is a solid state drive.
- Corrected a display issue on the Hyper-V host product. GUID was being displayed in lieu of the guest computer name.
- Clarified the code to ensure the user realized the schedules maximum duration only applies to the online portion of the schedule. There is not maximum duration for a boot-time defragmentation pass.
- Ensure that PerfectDisk does not attempt to perform an offline defragmentation pass on a FAT/FAT32 from a schedule. This functionality is not supported.
- Corrected some typo's in displayed text
- Changed the default behavior regarding file modification date on FAT/FAT32 drives. When no date exists we now display nothing. We used to display 1980.
- Improved the layout to better fit all the on screen data
- Corrected a bug when defragmenting drives greater than 2,147,483,648 clusters (8 TB with 4k cluster size)
- Improved boot-time defragmentation logging.
- Corrected a display issue with the schedule tab buttons when using Group Policy.
- Certain error messages were improved to aid the user in understanding what went wrong.
Console enhancements and fixes:
- Improved the task planner
- Corrected an infinite loop in the Console User Interface. PerfectDisk Enterprise Console would appear to hang.
- Corrected a crash in the Console user interface.
- Updated to SQLExpress Service pack 3.
- Displayed the Space explorer Client when PerfectDisk is not installed. This feature replies on PerfectDisk Client being present.
- Improved the historic data reports.
- Default Configuration profile data values have been adjusted to better represent the real world.
- Corrected Connect using PD option not being available if Hyper-V Host installed on Console computer.
- Corrected Console listing Hyper-V Host software as VMware.
- Corrected import of settings that created registry keys that were not correct type.
- Corrected Console reports that incorrectly did not include the Console computer.
- Corrected issue in Deployment wizard that incorrectly displayed local/GP schedules.
Get PerfectDisk and defrag news like this earlier — follow
PerfectDisk on Twitter.
Located in New Jersey, PS Systems is a certified dealer for most major hardware and software companies. The company is a single source for many companies' business automation needs. It works on office information systems, integrated automatic data collection through the use of bar coding, imaging and real-time data communications networks. The company maintains strategic alliances with numerous contractors, ISPs and network consultants.
Like most businesses these days, virtualization is an increasingly important piece of the every day work PS Systems does. Mike Packard is a software engineer at PS Systems, and like all good software engineers, he'd rather spend his days in a productive manner, programming and testing, rather than spending valuable time administering his systems to keep them performing optimally. And virtualization performance is an important piece of the puzzle.
Mike told us recently how he benefits from PerfectDisk 11 VMware Workstation Bundle:
"As a software engineer, I use virtual machines extensively to maintain a clean development platform and for repeatable testing. Before PerfectDisk VMware, I had to spend a lot of time defragging, shrinking, and maintaining each of my virtual machines. It was a constant effort and not fun.
"PerfectDisk VMware allows me to have very well-maintained virtual machines with a fraction of the time and effort. It lets me batch process all my machines in unattended maintenance after-hours. This means I can spend my day hours programming and testing, rather than running maintenance tasks. I'd say that PerfectDisk VMware is the high-strength aspirin for my virtual machine headaches. It really takes the pain away."
Virtualization defrag with PerfectDisk VMware Workstation Bundle for improved virtualization performance and recaptured time for yourself.
It's how to defrag a computer - virtual or physical.
Based on the testing we did with VMware's Scott Drummond, we recently published a new white paper,
The Perfomance Impact of Windows Guests Defragmentation and Free Space Consolidation on VMware ESX. The paper details the results of testing done to determine the effectiveness of defragmenting virtual servers.
The results, from testing done on a VMware ESX Cluster, indicate that defragmentation and free space consolidation of Windows server guests have a positive effect on the performance of both the Windows server guests and the host. While the testing was done on an ESX cluster, due to the nature of the NTFS file system, one could conclude that there would be similar impact across all virtualization performance, including vSphere performance and Hyper-V performance.
The white paper details various real statistics and numbers that we found in our testing. Of course, it is our testing only. We highly encourage IT administrators to do their own testing of defrag's effect on virtualization performance. But we are confident similar conclusions will be reached.
You can see the entire white paper
here.
Related Post:
Virtualization performance --Windows guest defragmentation
Scott Drummonds is the performance guy at VMware. All performance-related issues end up on his plate, and he spends much of his time analyzing data along with customer environments to determine how users can get the most out of what they have. From time to time, he's taken a look at disk defragmentation and its impact on virtualization performance.
We got together with Scott to evaluate the impact of guest defragmentation on a single virtual machine. In his widely-read blog, Scott sets up the test:
"Before I describe the test and its results, I want to share an important point on guest defragmentation. Most of the computer literate are aware that file fragmentation–the separation of logically contiguous pieces of a file–can hurt storage performance. But many may not realize that free space fragmentation is as big of an issue. When free space is fragmented, writes take longer and files are re-fragmented rapidly. PerfectDisk defragments files and free space and the results below benefit from both of these improvements."
I'm not going to give away the results here, but I will quote just one of Scott's comments here: "Let me repeat one of those amazing data points: the average IO latency dropped from about 55 ms to less than 4 ms. While this is a phenomenal number, the increase depends on characteristics of the storage system. Since these improvements are configuration dependent, your results may vary considerably."
You can read Scott Drummonds entire post, with lots of data and charts, here.
Get the lowdown on virtualization performance and virtualization defrag from VMware's top performance guy.
With the PerfectDisk 11 client solutions now rolled out, the next phase is on for our virtualization performance solutions:
- PerfectDisk 11 vSphere Bundle
- PerfectDisk 11 Hyper-V Bundle
- PerfectDisk 11 VMware Workstation Bundle.
As I’ve mentioned previously, we had the largest beta test ever for PerfectDisk 11, and had great feedback. For our vSphere defrag and Hyper-V defrag solutions, we had significant feedback that gave us reason to take some very big usability enhancements. The final phase of the PerfectDisk 11 beta test for these products begins this week.
Stay tuned to this space and Twitter for updates.
There are a few questions that often come up with regards to the PerfectDisk virtualization defrag solutions in the vSphere/ESX environments, versus the standard PerfectDisk server defrag.
1. What does PerfectDisk 10 for vSphere offer that PerfectDisk 10 Server does not?
Or in other words, what's the difference between performing a Windows Server 2008 defrag (or earlier server versions) and a virtualization defrag? There are various differences but the following two key points summarize the major points:
A. The vSphere-specific product offers an exclusive, patent-pending feature called "Virtualization Awareness" that allows file system optimization to take place without the contention for resources that normally takes place with automatic background defragmentation. This ensures that optimizations run dynamically without supervision or intervention, while never negatively impacting performance. It also offers integration features for VMware's vCenter and support for vMotion.
B. Licensing for the vSphere product is offered per physical host with volume pricing. This means unlimited Client/Guest licenses are provided once the physical host is licensed. This also means no consideration as to the number of Guests or even the number of CPUs needs to be taken. In the example of a vSphere environment with 10 instances of ESX Server 3.5/4.0 running, that would equate to 10 licenses of "PerfectDisk10 vSphere."
2. Does the vSphere product perform defragmentation on the ESX layer / VMFS? Does it install on the ESX Hosts or just the Guests?
No. VMFS does not support external file system optimization. Installation does not take place on the physical ESX host, a small agent is instead installed on each Windows based Guest.
3. Is any modification of the vSphere environment required in order to use PerfectDisk 10 vSphere?
No. The use of PerfectDisk10 vSphere does not require any modification to your existing configuration.
Smarter. Simpler. Priced for today's world.
The PerfectDisk 10 Enterprise Console has been updated to Build 127. This latest build provides enhanced support with a VIM plug-in to VMware vSphere Client 4.0 plus a few bug fixes.
The vSphere plug-in allows users to access the PerfectDisk 10 Enterprise Console directly from within the vSphere Client, providing a single point of management for their vSphere-based Windows guests. Using the PerfectDisk Enterprise Console plug-in, users can easily deploy, configure and schedule PerfectDisk, allowing for enhanced disk space management in vSphere environments, for vSphere defrag.
You can “Check For Updates” under Help in the PerfectDisk console, or wait for Autoupdate.
Get news like this earlier — follow PerfectDisk on Twitter.
I was recently invited to speak at a VMware User Group meeting about the impact of file and free space fragmentation in a virtual environment. Before I started the presentation I asked the audience how many had ever defragmented a physical server and about 30% raised their hand. Then I asked how many had ever defragmented a virtual server and about 10% raised their hand. This is a pretty typical response.
Systems administrators running virtual machines seem to overlook the fact that each virtual machine is running its own version of Windows Server and within that machine NTFS is busy fragmenting files and free space. While fragmentation is a performance-stealing problem on physical servers, the problem is compounded on virtual machines where each virtual instance is competing for a finite amount of resources. This leads to resource contention between Windows guests and, with Hyper-V, contention between the Windows guests and the host. It is essential to keep the files defragmented and the free space consolidated on Windows guests to maximize performance and minimize resource contention in a virtual world.
After my presentation one attendee told me his company was having performance issues with several of their virtual servers. They were looking at upgrading the hardware and bringing in a consultant. He knew these virtual machines were frequently updating files, the kind of activity that produces fragmentation, but he never thought of fragmentation as the problem. After sitting through my presentation he was convinced fragmentation was the culprit and we just saved them a lot of unnecessary expense. In complex technical environments it is easy to overlook the obvious when looking for a solution. Sometimes you just need to defrag a computer to get a lot more out of it, and that includes VM's.
In a report by editor-at-large Charles Babcock, Information Week recently reported on IBM's plans to bring the virtual machines of multiple vendors under one management umbrella called IBM VMControl. The new VMControl is due out in December, and no, it won't include any defragmenting PC or server defrag software. But, from a single management console, it addresses a new data center reality - VM sprawl (as opposed to the old physical server sprawl).
So it is with the approach we are taking with PerfectDisk as we move beyond hard drive defragmentation to the new virtual world. We've certainly learned a lot and continue to learn since we started moving in this direction. We've built up our virtualization expertise and continue to do so, both internally and through alliances and partnerships outside Raxco.
On the data center side, PerfectDisk has evolved from its initial ESX offering to more comprehensive solutions for the new vSphere 4 as well as Microsoft's Hyper-V environment. This is in addition to our disk defragmentation tool for VMware Workstation and VMware Server. Now, through the PerfectDisk 10 Enterprise Console, this more heterogeneous environment can be managed and controlled, and there will be additional capabilities added with PerfectDisk 11.
In his article, Babcock reports numbers in the 60% - 80% range for IBM customers using multiple hypervisors. The trend is clear for this still-evolving world of virtualization, and it carries over to virtualization defrag as well. The data center is moving well beyond a simple server defrag. Physical and virtual servers need to be addressed, as do guests and hosts, and all from multiple vendors. With our patent-pending Virtual Awareness technology, we're working hard to build on the foundation we've started, and move forward with our customers, as their virtualization partner.
To help manage the sprawl.
With VMware's recent release of VMware Workstation 7, support is immediately available for PerfectDisk 10 VMware Workstation customers. 7 introduces many enhancements, including installing and running VMware vSphere 4 and VMware ESXi in a virtual machine.
VMware calls VMware Workstation "the gold standard in desktop virtualization." So it makes sense for those that want disk defragmenter software to work seamlessly with VMware products, they look to the gold standard in virtualization defrag.
PerfectDisk 10 VMware Workstation incorporates and automates the disk space-saving “Shrink” and re-indexing found in VMware Workstation, enhancing their effectiveness. It defragments Windows guests and moves all the free space to the end of the disk, allowing the Shrink utility to maximize the amount of valuable disk space it can recover.
The defragment program for automatic defragmentation, re-indexing and shrinking of VMware Workstation - it's only here.
With our recent announcement of support for vSphere 4, PerfectDisk now provides unparralled support for vSphere defrag. VMware has received a great deal of interest with this cloud operating system, as its battle with Microsoft's Hyper-V, Citrix' XenServer and others shows no sign of abating, but only intensifying. While auto defrag is a key element of any solution for a virtualization defrag tool, any defragmenter program that fails to address the unique complexities of a virtual environment will come up short.
Because vSphere 4 is able to manage large pools of virtualization computing infrastructure, including hardware and software, its overall performance is critical to organizations implementing it. This is one area where a defragmenter program can help. And with its ability to defrag seamlessly in a vSphere environment, providing the patent-pending Virtual Awareness defragmentation that eliminates resource contention and ensures efficient use of resource, PerfectDisk 10 vSphere ESX delivers.
Time to defrag VMware environments like you mean it...with true integration.
Continuing PerfectDisk’s pioneering role in defragmentation for virtual environments, I was happy to announce support for VMware vSphere 4 in the latest PerfectDisk 10 build. We’ve established a great relationship with VMware over the past few years, and as part of this we continue to evolve our support for VMware offerings. Whichever side of the ESX/Hyper-V — or others — debate you come down on, we know defragmentation of these environments is an important piece of getting these systems running at peak performance and efficiency, making the best use of resources while eliminating resource contention.
Upon release, VMware dubbed its vSphere 4 as the industry’s first operating system for building the internal cloud, enabling the delivery of efficient, flexible and reliable IT as a service, and bringing cloud computing to enterprises in an evolutionary way. As the leader in virtualization defrag, it’s a natural for PerfectDisk to lead the way in support for vSphere 4 and vSphere defrag.
Head to the clouds, and remember to defrag -- virtual defrag.
Powered by Compendium Blogware