Active vs. Pa.s.sive Profiling Xenoprofile supports both active and pa.s.sive modes for domain profiling.

When profiling in pa.s.sive mode, the results indicate which domain is running at sample time but don"t delve more deeply into what"s being executed. It"s useful to get a quick look at which domains are using the system.

In active mode, each domU runs its own instance of OProfile, which samples events within its virtual machine. Active mode allows better granularity than pa.s.sive mode, but is more inconvenient. Only paravirtualized domains can run in active mode.

Active Profiling Active profiling is substantially more interesting. For this example, we"ll use three domains: dom0, to control the profiler, and domUs 1 and 3 as active domains.

0#opcontrol--reset 1#opcontrol--reset 3#opcontrol--reset First, set up the daemon in dom0 with some initial parameters: 0#opcontrol--start-daemon--event=GLOBAL_POWER_EVENTS:1000000:1:1 --xen=/boot/xen-syms-3.0-unstable --vmlinux=/boot/vmlinux-syms-2.6.18-xen0--active-domains=1,3 This introduces the --xen --xen option, which gives the path to the uncompressed Xen kernel image, and the option, which gives the path to the uncompressed Xen kernel image, and the --active-domains --active-domains option, which lists the domains to profile in active mode. The option, which lists the domains to profile in active mode. The :1 s :1 s at the end of the event option tells OProfile to count events in both users.p.a.ce and kernel s.p.a.ce. at the end of the event option tells OProfile to count events in both users.p.a.ce and kernel s.p.a.ce.

NoteSpecify domains by numeric ID. OProfile won"t interpret names.

Next, start OProfile in the active domUs. The daemon must already be running in dom0, otherwise the domU won"t have permission to access the performance counters.

1#opcontrol--reset 1#opcontrol--start Run the same commands in domain 3. Finally, begin sampling in domain 0: 0#opcontrol--start Now we can run commands in the domains of interest. Let"s continue to use the kernel compile as our test workload, but this time complicate matters by running a disk-intensive benchmark in another domain.

1#timemakebzImage 3#timebonnie++ When the kernel compile and Bonnie++ have finished, we stop OProfile: 0#opcontrol--stop

0#opcontrol--shutdown 1#opcontrol--shutdown 3#opcontrol--shutdown Now each domU will have its own set of samples, which we can view with opreport opreport. Taken together, these reports form a complete picture of the various domains" activity. We might suggest playing with the CPU allocations and seeing how that influences OProfile"s results.

An OProfile Example Now let"s try applying OProfile to an actual problem. Here"s the scenario: We"ve moved to a setup that uses LVM mirroring on a pair of 1 TB SATA disks. The hardware is a quad-core Intel QX6600, with 8GB memory and an ICH7 SATA controller, using the AHCI driver. We"ve devoted 512MB of memory to the dom0.

We noted that the performance of mirrored logical volumes accessed through xenblk xenblk was about one-tenth that of nonmirrored LVs, or of LVs mirrored with the was about one-tenth that of nonmirrored LVs, or of LVs mirrored with the --corelog --corelog option. Mirrored LVs with and without option. Mirrored LVs with and without corelog corelog performed fine when accessed normally within the dom0, but performance dropped when accessed via performed fine when accessed normally within the dom0, but performance dropped when accessed via xm block-attach xm block-attach. This was, to our minds, ridiculous.

First, we created two logical volumes in the volume group test test: one with mirroring and a mirror log, and one with the --corelog --corelog option. option.

#lvcreate-m1-L2G-ntest_mirrortest #lvcreate-m1--corelog-L2G-ntest_coretest Then we made filesystems and mounted them: #mke2fs-j/dev/test/test*

#mkdir-p/mnt/test/mirror #mkdir-p/mnt/test/core #mount/dev/test/test_mirror/mnt/test/mirror Next we started OProfile, using the --xen --xen option to give the path to our uncompessed Xen kernel image. After a few test runs profiling various events, it became clear that our problem related to excessive amounts of time spent waiting for I/O. Thus, we instruct the profiler to count option to give the path to our uncompessed Xen kernel image. After a few test runs profiling various events, it became clear that our problem related to excessive amounts of time spent waiting for I/O. Thus, we instruct the profiler to count BUS_IO_WAIT BUS_IO_WAIT events, which indicate when the processor is stuck waiting for input: events, which indicate when the processor is stuck waiting for input: #opcontrol--start--event=BUS_IO_WAIT:500:0xc0 --xen=/usr/lib/debug/boot/xen-syms-2.6.18-53.1.14.el5.debug --vmlinux=/usr/lib/debug/lib/modules/2.6.18-53.1.14.el5xen/vmlinux --separate=all Then we ran Bonnie++ on each device in sequence, stopping OProfile and saving the output each time.

#bonnie++-d/mnt/test/mirror #opcontrol--stop #opcontrol--save=mirrorlog #opcontrol--reset The LV with the corelog displayed negligible iowait, as expected. However, the other experienced quite a bit, as you can see in this output from our test of the LV in question: #opreport-t1--symbolssession:iowait_mirror warning:/ahcicouldnotbefound.

CPU:Core2,speed2400.08MHz(estimated) CountedBUS_IO_WAITevents(IOrequestswaitinginthebusqueue)withaunitmaskof0xc0(All cores)count500 ProcesseswithathreadIDof0 ProcesseswithathreadIDof463 ProcesseswithathreadIDof14185 samples%samples%samples%appnamesymbolname 3291.42861593.750000xen-syms-2.6.18-53.1.14.el5.debugpit_read_counter 12.85710000ahci(nosymbols) 12.85710000vmlinuxbio_put 12.85710000vmlinuxhypercall_page Here we see that the Xen kernel is experiencing a large number of BUS_IO_WAIT BUS_IO_WAIT events in the events in the pit_read_counter pit_read_counter function, suggesting that this function is probably our culprit. A bit of searching for that function name reveals that it"s been taken out of recent versions of Xen, so we decide to take the easy way out and upgrade. Problem solved-but now we have some idea why. function, suggesting that this function is probably our culprit. A bit of searching for that function name reveals that it"s been taken out of recent versions of Xen, so we decide to take the easy way out and upgrade. Problem solved-but now we have some idea why.

Used properly, profiling can be an excellent way to track down performance bottlenecks. However, it"s not any sort of magic bullet. The sheer amount of data that profiling generates can be seductive, and sorting through the profiler"s output may take far more time than it"s worth.

[59] Excluding top(1), of course. Excluding top(1), of course.

Conclusion So that"s a sysadmin"s primer on performance measurement with Xen. In this chapter, we"ve described tools to measure performance, ranging from the general to the specific, from the hardware focused to the application oriented. We"ve also briefly discussed the Xen-oriented features of OProfile, which aim to extend the profiler to multiple domUs and the hypervisor itself.

Chapter11.CITRIX XENSERVER: XEN FOR THE ENTERPRISE

Until now, we"ve focused exclusively on the open source version of Xen. However, that"s not the only choice available. The Xen guys have also released a packaged version of Xen that"s aimed at turning Xen into a product that is suitable for the enterprise.[60]

One implication that you may have gotten reading between the lines of this book is that Xen is still in a state of development. It is, in a word, hackerware. It"s good "ware-obviously we think it"s stable enough for real people to use every day-but it"s still a lot of work to set up and get going.

There are a couple of reasonable explanations for this state of affairs. Partly this is because hackers are not, as a group, very good at finishing things. The work to take a product from 90 percent finished to 100 percent finished isn"t usually difficult, just tedious. Open source is wonderful, but it"s not great at producing full-on commercial products products.[61] Another issue is that Xen, by nature, is invasive and fundamental. One hesitates to use words like Another issue is that Xen, by nature, is invasive and fundamental. One hesitates to use words like paradigm paradigm, but there you have it-virtualization is a different way of computing, a different way of thinking about computers, and it requires a lot of very polished software support.

Citrix (who acquired XenSource, the company founded by the original Xen team) works to ease the transition by providing this software-creating a software stack, developing a certification process, and establishing best practices so that administrators can roll out Xen with a minimum of fuss and uncertainty. They work on the open source version of Xen and contribute changes to it, but they also do an additional level of QA aimed at turning Xen into a product that you can feel comfortable trusting with your business.

You may ask how this is possible, considering that Xen is still under the GPL. Citrix can do this while obeying the terms of Xen"s license because Xen"s client/server architecture and modular design allow them to extend the basic hypervisor, adding new features as modules and users.p.a.ce processes that work in conjunction with the GPL software. Citrix uses the open source hypervisor with open source Linux, plus added modules and proprietary control software to provide an integrated distribution of Xen, much like a traditional Linux distro but with a strong emphasis on virtualization.

Citrix"s Xen Products The Citrix product consists of two components, XenServer and XenEssentials. XenServer is the hypervisor and basic management tools, and it is available for free.[62] XenEssentials is a suite of utilities that cost money. XenEssentials is a suite of utilities that cost money.

The basic free product is simply called XenServer. XenServer supports most of the features of the paid Citrix products, with the same management interface. It"s aimed at development, test, and noncritical production deployments, as well as people who want to test or play with Xen.

Citrix"s pay product is called Citrix Essentials for XenServer, with various levels of licensing. It doesn"t have all the features of open source Xen, but it has all the features Citrix feels comfortable supporting and some commercial product exclusives. Citrix, as far as we can tell, charges as much as the market will bear for this version.[63] This is, of course, subject to change at any time. Negotiate with your Citrix representative, preferably in some form of gladiatorial combat. This is, of course, subject to change at any time. Negotiate with your Citrix representative, preferably in some form of gladiatorial combat.[64]

By and large, we"re going to focus exclusively on the base product and components that are available for free.

[60] Sorry, we know, marketing speak. But it is the easiest way for us to convey the aim of the product. Sorry, we know, marketing speak. But it is the easiest way for us to convey the aim of the product.

[61] This is not to disparage the fine work of the people behind polished products we use daily, such as Linux, Mozilla, and Vim. We"re just saying that the last 10 percent is the most difficult, not that it never gets done. This is not to disparage the fine work of the people behind polished products we use daily, such as Linux, Mozilla, and Vim. We"re just saying that the last 10 percent is the most difficult, not that it never gets done.

[62] That"s free as in beer, as the greybeards say. That"s free as in beer, as the greybeards say.

[63] We were unable to find coherent pricing information. We were unable to find coherent pricing information.

[64] After spending weeks trying to get prices for colocation out of salespeople, Luke suspects gladiatorial combat would be more pleasant than the traditional methods for negotiating price. prgmr.com favors the "the price on the website is the price you pay" model. After spending weeks trying to get prices for colocation out of salespeople, Luke suspects gladiatorial combat would be more pleasant than the traditional methods for negotiating price. prgmr.com favors the "the price on the website is the price you pay" model.

The Benefits of Using Citrix XenServer The XenServer product improves on open source Xen primarily in the area of manageability. They"ve streamlined and automated common tasks while retaining most of the transparency of open source Xen.

Ten Minutes to Xen Our model is one where the CD enters the drive and the computer is a better machine as a result (in ten minutes or less). That is what XenExpress[65] is all about. is all about.-Frank Artale, XenSource XenSource One of the best demonstrations of this is in what Citrix calls Xen in ten minutes Xen in ten minutes or or Ten to Xen Ten to Xen. They"ve dramatically simplified the bootstrap aspect of Xen, where you have to install a dom0 OS and modify it to work nicely with Xen"s control software and the hypervisor.

Citrix reasons that you shouldn"t actually be doing doing anything with the dom0, other than controlling domUs. Therefore, the product installs a basic Linux OS that includes only the components needed to run Xen: a kernel, a sh.e.l.l, some libraries, a text editor, Python, syslog, SSH (and so forth), and the Xen software. In this approach, software that"s not needed to control Xen, such as the daemons that provide a server"s anything with the dom0, other than controlling domUs. Therefore, the product installs a basic Linux OS that includes only the components needed to run Xen: a kernel, a sh.e.l.l, some libraries, a text editor, Python, syslog, SSH (and so forth), and the Xen software. In this approach, software that"s not needed to control Xen, such as the daemons that provide a server"s raison d"etre raison d"etre, should be installed in a domU. Of course, it"s still Linux-based on CentOS, in fact-and there"s nothing to stop you from installing other software. However, we recommend sticking with Citrix"s approach and keeping your core virtualization server uncluttered.

The basic package does, in fact, take about 10 minutes to install, as advertised. Be sure to get the supplementary Linux pack, which includes Debian templates and supporting tools for Linux VMs. When that"s done, it"s a simple matter to create domUs from the included Debian template or from install media.

Citrix XenServer has other advantages. Perhaps most important, it feels much more centralized than the open source Xen. All of the decisions that we"ve been writing about-storage, networking, and so forth-are handled in a centralized way, using a consistent interface. Where possible, they"ve made sensible default decisions for you. They"re not necessarily going to be the best for all situations, but at least they"ll be reasonable for Xen"s purposes.

Take storage, for example. Citrix uses the same architecture as open source Xen, using unmodified Linux drivers in the dom0 to access physical devices. They layer LVM on top of this to abstract physical storage and increase flexibility, as we"ve outlined elsewhere. Citrix builds on these open source tools by offering a way of administering storage through the same GUI as the more Xen-specific aspects of the system, allowing you to focus on virtual machines rather than obscure disk-administration commands. If you like, you can still use the familiar commands. Citrix"s Xen product isn"t out to reinvent the wheel or obfuscate the basic workings of the system; they"ve just provided an alternative to make common tasks a little easier.

[65] XenExpress was the name of the free product when XenSource was XenSource, before Citrix bought them. XenExpress was the name of the free product when XenSource was XenSource, before Citrix bought them.

The Disadvantages of Using Citrix XenServer Even with the high-end Essentials product, there"s a trade-off between stability and features. Citrix exposes only those hypervisor features that they feel are mature enough to use in a production environment. For example, migration was added a couple of years after it had been introduced in the open source version.

There"s also no easy way of moving VMs between open source and commercial Xen at the moment. (You can, of course, move VMs manually by using the lower-level methods outlined in Chapter9 Chapter9.) If you standardize on open source or commercial Xen, it may be difficult to reverse that decision later, although the Open Virtualization Format (OVF), which has some support from open source tools,[66] promises to improve the situation. promises to improve the situation.

Beyond that, open source is still an ideological issue. Some people use it whenever possible; some avoid it as a pestilence. We use the open source product because it"s good enough for us and because apparently our time is worthless. Citrix offers a straightforward transaction: Give them money and they"ll give you Xen as a product, rather than Xen as heavily customizable hackerware. Step on board and take your chances.

[66] is a good place to start. is a good place to start.

Getting Started Having said all that, the best way to get started with Citrix"s XenServer is probably just to try the product and see if you like it. The entry-level version is available for free. You can download it at and upgrade it at any time simply by entering a license key. Besides, they"re telling the truth when they say it takes about 10 minutes to install, so why not? and upgrade it at any time simply by entering a license key. Besides, they"re telling the truth when they say it takes about 10 minutes to install, so why not?

Prerequisites First, check to make sure that you meet the minimum system requirements: 64-bit CPU, that is, AMD Opteron, Athlon 64, Phenom, or whatever else AMD"s marketing department has come up with, or most Intel Xeons of the last few years, as well as the Core 2 (but not Core).

A certain amount of memory, depending on how many virtual machines you want. Citrix"s minimum is 1GB, which sounds reasonable to us.

Sufficient disk s.p.a.ce. The XenServer backend will take 8GB, leaving the rest available for domUs. Of course, you can also use network storage for VMs.

HVM support is required if you want to run Windows domUs, but otherwise it"s optional.

Installing Citrix XenServer As we"ve mentioned, Citrix"s product is a complete system; install it like any other OS. For us, that meant downloading the ISO, burning it to a CD, and booting our target machine from the CD. We also grabbed the Linux Guest Support disc Linux Guest Support disc, which includes support for Linux guests.

The machine goes through a nongraphical install and asks some standard questions about keyboard, time, and network setup-the usual. Compared to a normal Linux install, it"s incredibly spare and streamlined because the product itself has the single focus of virtualization. For example, there"s no opportunity to set up part.i.tioning. At the end, it prompted us to insert supplementary CDs, so we put in the Linux support disc.

Ten minutes and one reboot later, we"re staring at a screen that advises us to log in via the administration frontend, XenCenter.

USING A SERIAL CONSOLE WITH XENSERVERWe would never consider using a server without serial console access. Citrix"s Xen product, although it doesn"t support a serial console out of the box, can support a serial console with a little configuration.It"s a little more difficult than you"d expect because Citrix uses Extlinux to boot rather than GRUB. However, Extlinux"s configuration is similar. The only file we need to adjust is /boot/extlinux.cfg /boot/extlinux.cfg. Note that we specify the options to Xen and the Linux kernel on the same long line:SERIAL0115200

defaultxe

prompt1

timeout50

labelxe

#XenServer

kernelmboot.c32

append/boot/xen.gzdom0_mem=752Mlowmem_emergency_pool=16M =115200,8n1console=com1--- /boot/vmlinuz-2.6-xenroot=LABEL=root-jhawazvhro console=ttyS0,115200n8---/boot/initrd-2.6-xen.imgBecause this is basically CentOS, it already has ttyS0 listed in /etc/inittab /etc/inittab with a getty. Reboot and enjoy the serial console. with a getty. Reboot and enjoy the serial console.

Citrix"s Xen GUI: XenCenter We are, of course, fond of following advice.

Citrix"s system, like the open source version of Xen, uses a client/server architecture to control the virtual machines. Unlike the open source version, they include a graphical Xen console that can automate many of the boring details of running Xen.[67]

In fact, the package includes both a GUI and a command-line utility. As of version 5, the GUI is a Windows application called XenCenter, and the command-line tool is called xe xe. They offer roughly the same functionality, but the XenCenter GUI has more features, and xe xe supports certain more arcane operations. Citrix suggests using supports certain more arcane operations. Citrix suggests using xe xe for scripted (or otherwise automated) operation and the XenCenter for interactive administration. There"s also a character-based menu system called for scripted (or otherwise automated) operation and the XenCenter for interactive administration. There"s also a character-based menu system called xsconsole xsconsole that normally runs on the Xen server"s physical console, but which can be run in any sh.e.l.l session in dom0. It provides access to many common operations. that normally runs on the Xen server"s physical console, but which can be run in any sh.e.l.l session in dom0. It provides access to many common operations.

You"ll need a Windows machine for the GUI client. Although version 3.2 and previous versions were written in Java, and are therefore cross-platform, versions 4.0 and above require the .NET-based XenCenter. Previous versions of the client will not be able to connect to the XenServer host. The requirement that the client runs under Windows, of course, also means that you can"t run the client directly on the machine that"s running Citrix"s product.[68]

Unlike the open source version of Xen, communication between the tools and hypervisor doesn"t go through xend xend. Instead, both the command-line and graphical tools connect to the xapi xapi service on the Xen server via TCP/IP, using SSL to encrypt traffic. service on the Xen server via TCP/IP, using SSL to encrypt traffic.

[67] The XenCenter cannot connect to open source Xen. We tried. The XenCenter cannot connect to open source Xen. We tried.

[68] Although you could run the client inside of a Windows install under the XenSource product, it does raise an interesting chicken-and-egg problem. Although you could run the client inside of a Windows install under the XenSource product, it does raise an interesting chicken-and-egg problem.