PowerCLI: Configure iSCSI one-liner

While migrating a small environment to vSphere today I ran into my nemesis Host Profiles again. When are they going to Fix these things? The fact that they are incapable of even rudimentary iSCSI configuration is embarrassing. I’m sure vmware will fix it, but until then I wrote a simple one-liner that will configure iSCSI on a new host.

~Glenn

Experimenting with the vSphere ESXi install process

I suppose easy is relative.

One of the comments to the post I made about touch free ESXi installs asked about testing without having to reboot and wait for the install process to load, and fail, to determine what went wrong. I did this testing by switching to a different console and using the Python interactive shell to load the same modules VMware uses, call their methods, and simply look at the values returned. By reading their code I was able to determine what the dialogs (the prompts presented to the admin during the install process) return, and the simply return that without the dialog occurring.

From the install screen you can switch to a different console (Alt+F1, if I recall correctly), and then access a command line (you may need to use the “unsupported” trick to get the command line).

The install process is actually quite interesting. VMware is booting, via ISOLINUX, into a full ESXi environment (instead of the standard ESXi yellow and black, they display the install dialogs), asking you which disk you want to use, then formatting that disk with VMFS and copying a virutal machine to the new VMFS volume. They then configure the boot partition to start the ESX kernel and start the ESXi management virtual machine. It’s rather interesting (well, to me) what they are able to accomplish because of the power and flexibility having such a small hypervisor affords them. Not to mention the sheer genius of using their own hypervisor to perform the install of itself…simplicity!

Read moreExperimenting with the vSphere ESXi install process

PowerCLI: Apply-VMHostProfile passing parameters via $AdditionalConfiguration

I’ve ran across this particular issue myself, and submitted a bug to the PowerCLI team, but shortly after Andrew posted his ESXi 4.0 autoinstall Tim asked about this very issue.  There is a documentation error in Example #5 from the Apply-VMHostProfile cmdlet help.  Which contains the following code example.

Sadly if you tried to execute the above you would get the following error.

At first this may appear a little cryptic, but it get’s a lot clearer once we inspect the object types in use.

The example from the help docs was apparently expecting a Hashtable to be returned from apply-VMhostProfile.  Instead we found an array of DictionaryEntry objects… hence the error.

There are two possible work around’s we can employ until the PowerCLI team ships a fix.  The first one is complicated, but dynamic.

<pre class=’PowerShellColorizedScript’><span style=’color:#ff4500′>$profile</span> <span style=’color:#a9a9a9′>=</span> <span style=’color:#0000ff’>Get-VMHostProfile</span> <span style=’color:#000080′>-Name</span> <span style=’color:#8a2be2′>testProfile</span>

<span style=’color:#ff4500′>$additionalConfiguration</span> <span style=’color:#a9a9a9′>=</span> <span style=’color:#0000ff’>Apply-VMHostProfile</span> <span style=’color:#000080′>-ApplyOnly</span> <span style=’color:#000080′>-Profile</span> <span style=’color:#ff4500′>$profile</span> <span style=’color:#000080′>-Entity</span> <span style=’color:#8a2be2′>10.23.114.166</span>
<span style=’color:#000000′>(</span><span style=’color:#ff4500′>$additionalConfiguration</span> <span style=’color:#a9a9a9′>|</span> <span style=’color:#0000ff’>Where-Object</span> <span style=’color:#000000′>{</span><span style=’color:#ff4500′>$_</span><span style=’color:#a9a9a9′>.</span><span style=’color:#000000′>Name</span> <span style=’color:#a9a9a9′>-eq</span> <span style=’color:#8b0000′>’network.hostPortGroup[“key-vim-profile-host-HostPortgroupProfile-VMkernel”].ipConfig.IpAddressPolicy.address'</span><span style=’color:#000000′>}</span><span style=’color:#000000′>)</span><span style=’color:#a9a9a9′>.</span><span style=’color:#000000′>Value</span> <span style=’color:#a9a9a9′>=</span> <span style=’color:#8b0000′>’10.0.0.128′</span>
<span style=’color:#000000′>(</span><span style=’color:#ff4500′>$additionalConfiguration</span> <span style=’color:#a9a9a9′>|</span> <span style=’color:#0000ff’>Where-Object</span> <span style=’color:#000000′>{</span><span style=’color:#ff4500′>$_</span><span style=’color:#a9a9a9′>.</span><span style=’color:#000000′>Name</span> <span style=’color:#a9a9a9′>-eq</span> <span style=’color:#8b0000′>’network.hostPortGroup[“key-vim-profile-host-HostPortgroupProfile-VMkernel”].ipConfig.IpAddressPolicy.subnetmask'</span><span style=’color:#000000′>}</span><span style=’color:#000000′>)</span><span style=’color:#a9a9a9′>.</span><span style=’color:#000000′>Value</span> <span style=’color:#a9a9a9′>=</span> <span style=’color:#8b0000′>’255.255.255.0′</span>

<span style=’color:#0000ff’>Apply-VMHostProfile</span> <span style=’color:#000080′>-ApplyOnly</span> <span style=’color:#000080′>-Profile</span> <span style=’color:#ff4500′>$profile</span> <span style=’color:#000080′>-Entity</span> <span style=’color:#8a2be2′>10.23.114.166</span> <span style=’color:#000080′>-Variable</span> <span style=’color:#ff4500′>$additionalConfiguration</span></pre>

I actually don’t like this approach even though it’s a modified version of the included example.  I prefer just a simple static Hashtable.

All in all, the HostProfile cmdlets are surprisingly complete, and I think the majority of the “issues” I’ve ran across are a result of the SDK itself.  The Host Profiles sections of the API just don’t have the same fit and finish I’ve come to expect in a VMware API.

I’m sure carter and team will have this fixed in the next release, untill then…

~Glenn

ESXi 4.0 autoinstall

Being, first and foremost, lazy and getting my paychecks for being a system administrator, I felt that the amount of work involved in loading ESXi 4.0 on my blades was entirely too much. I have well over 100 blades, each one needing to have vSphere loaded onto it, configured, and added to vCenter. Even using the directions scattered across the internet about reducing the amount of effort involved in loading vSphere was too much for me.

Others have documented how to PXE boot ESXi elsewhere on the internet, however I wasn’t interested in having a “stateless” install…I merely wanted to automate installing ESXi to the local hard drive. My blades have a single hard drive, a single generation one SSD or two SAS drives in a RAID 1 depending on the vendor, and I simply want the installer to always install to that drive without bothering me. Loading from the “remote media” functionality of the DRAC/iLO for the blades takes forever, so I wanted to be able to install using PXE and push the media over that medium.

So, having been a developer for several years I decided to dive further into the the install process than others had detailed. Turns out that eliminating all input from an administrator to load the operating system was pretty simple.

The end result is that I am able to power on a blade, hit F12 to have it PXE boot and walk away. Some time later, we can use PowerShell and the PowerCLI to find the hosts (they will be somewhere in the DHCP scope of the provisioning LAN), give them a permanent IP and hostname, then configure them and add them to vCenter. By using PXE and the interactionless (yes, I did make up that word) install, I cut the time to load ESXi from about 45 minutes (using the remote media function takes FOREVER!) to less than 10.

Read moreESXi 4.0 autoinstall

PowerCLI: Remove SMVI snapshots

I wrote this script about a year ago to deal with errant SMVI snapshots, and was drafting this blog post when my rss feed caught me off guard. It appears Matt Robinson has beat me to the punch line.   He has produced a Perl script that cleans up any leftover snapshots, but if you favor a PowerShell approach… I give you Remove-SMVISnapshots.

Read morePowerCLI: Remove SMVI snapshots

PowerCLI: Find vCenter without vCenter

If you don’t know already PowerCLI now has two modes single and multiple.  It stands for exactly what you think it does.  In single mode when you execute a command PowerCLI runs that command against the server you’re connected too.  Multiple mode allows you to specify multiple vCenter/ESX/vSphere host, and when you execute a command it runs that command against every server you’ve specified! This had to be one painful feature to get right, but the PowerCLI team nailed it.

I’ll admit when I first played with it I thought I would never need/use multiple mode.  That is until our vCenter server was inadvertently shutdown instead of rebooted.  Normally this would lead to one of two out comes.
A.) forcefully register vCenter on the first host I hit and power it up.
B.) A twenty minute search for the host that has vCenter.

Well today I didn’t feel like doing either… On a whim I tried this new-fangled multiple connection thing… IT WORKED!

I’ve since wrapped all this up in a batch file and added it to our playbook for a lights out recovery of virtual center!

~Glenn

Industrial strength vCenter database maintenance (for SQL Express too!)

Over time, vCenter does a huge number of reads and writes to it’s database. Most of these come from the performance stats, and depending on the level of stats you have turned on, the number of writes can be very substantial. After a while, even with a small-to-medium infrastructure, the database will need some love to keep it working like it should. VMware has some recommendations about maintenance on your database, specifically here if you are using SQL Server. However, their recommendations only do a little to help if you have a very badly fragmented database.

You see, dbcc indexdefrag only defragments the index. This means that it will reorganize the pages in the index so that they are near to each other, but if you have a lot of partially filled pages, it still won’t do a whole lot of good. Don’t misunderstand me, it will help as it means that an index scan will perform significantly better since the scan will not require as much I/O work, but there are usually very few pages freed, which means that there could still be a large number of (potentially unnecessary) pages in the index.

Here is the example output that VMware provides in the KB article linked above:

Notice the “Avg. Page Density (full)” item…each page is only 36% full, this means that there is a huge amount of white space in each page, which under the right circumstances, can be ok, but in this case, we probably don’t want that. Generally speaking, if there is a high number of writes to the table (an consequentially the index) then you want a lower page density (a.k.a. fill factor), a higher number of reads would mean a higher fill factor is appropriate.

vCenter, by my estimation, does quite a few writes to the database, but it also does a fair amount of reads (every time you view the performance tab, every time those graphs refresh, it’s a series of reads). Additionally, the roll-up stats act as a bit of an equalizer, as they do a significant amount of deletions in the tables.

What all this means, to me anyway, is that a medium-high fill factor (70-80%) seems to fit the bill. So, 36% suddenly seems quite low, and that 36% is only marginally improved with the index defrag.

Well, there happens to be a command available that rebuilds the indexes: alter index rebuild. What this means is that the current indexes (and all associated pages on disk) are deleted, after which the database engine will read each row in the table and rebuild the index, making it contiguous on disk, and we can specify the fill factor to what we think is appropriate.

Fortunately, some kind soul over on MSDN posted a very nice script that will check the status of all indexes on all tables in a particular database. To use the script, open SQL Server Management Studio (SSMS), connect to your SQL Server instance and open a new query window. Enter a use statement for the database:

Then copy the SQL in the post made by N8WEI into the SSMS query window below the above USE statement. There are a couple of parameters that you can set: the fill factor, the index defrag threshold, the index rebuild threshold, and the “report only” value. Fill factor is just that, I set it to 70%. The index defrag threshold (“reorg_frag_thresh”) can stay relatively low (10% or less), the rebuild threshold (“rebuild_frag_thresh”) can stay at 30% (unless you want to force a rebuild on all indexes, in which case set reorg_frag_thresh to 1 and rebuild_frag_thresh to 2). The last option (“report_only”) determines if the script will actually take any action. Leave it set to 1 for the moment.

By leaving report_only to 1, it will only check the indexes, it will not perform any action upon them. So when you press the “Execute” button (or F5), it will scan through the indexes for the database and check their fragmentation level. The result window will have a list of the indexes and the action to be taken.

Before you proceed, be aware of a couple of things. Doing an index defrag is transparent to the users (e.g. vCenter). If the database is very busy, it may seem a little slow while the defrag is occurring, but the database will still be usable. When an index rebuild is performed, the table is locked so that no action can be taken during the operation. This means that for the duration of the rebuild the table will be unavailable for read/write. This probably isn’t a big deal…I run the operation about once a month on all tables and it takes less than 60 seconds to complete. This means each table is only locked for a few seconds, at most, which is probably ok. vCenter updates each host’s performance stats every 20 seconds, and it shouldn’t fail the insert operation if it has to wait a second or two. If you are concerned about it, simply shutdown the vCenter service on the server and it will stop performing queries.

With that said, change report_only to 0 (zero), and hit the button. It will do it’s thing (the first time may take a minute or two depending on your stats level), and when you’re done your indexes should be fresh and new.

If you want a warm and tingly, be sure to run the VMware recommended dbcc showcontig on the history tables before and after executing the index rebuild(s). You should see near 99.9%+ scan density and a page density within a few hundredths of a percent of the value specified.

The end result is that performance is significantly improved, even for operations that don’t involve the performance stats (though those are especially noticeable). I run the above script, with settings to rebuild indexes, once a month. I also run the script with settings to defrag the indexes once a day, which helps to prevent as many page splits and, consquentially, leads to less fragmentation. (Hint: Use the SQL Server Agent to schedule the operations.)

Oh, and this works with SQL Express, though you may have to turn on connecting via TCP before you can use SSMS to execute queries (and there’s no SQL Server Agent, so you’ll have to use scheduled tasks).

And, if you are using SQL Express, you can also use SSMS Express (even if you are using the full SQL Server you can use SSMS Express). The SQL Server Management Stuido 2008 Express is a very nice (and very functional) application, and it’s freely available from Microsoft (::watches for sky to begin falling::).

vCenter 2.5 and 2048bit RSA keys

So, it seems that prior to vCenter 4.0 it is not possible to use an RSA key length longer than 1024 bits for SSL.

I came to this discovery because for the last few days I’ve been replacing the default SSL certs for each of the ESX (3.5u4) hosts with certs that are signed by our local CA (and therefore trusted automatically). Our security office requires a key length of 2048 bits. Nothing less.

Since I wasn’t really paying attention to the documentation that VMware has provided (here, here, here, here and here), I missed where they actually told me it wouldn’t work.

So, when I replaced the cert and tried reconnecting my hosts I got an error.

Not being one to be deterred by “a database schema limitation”, I decided to fix the problem. The kb article here tells us what the error is: “string too large for database”, and there is also some references to the password’s inability to be decrypted.

In my test environment I decided to do a test…modifying the database schema so that it can hold the encrypted password in the field. The default schema (for vCenter 2.5 anyway) has the VPX_HOST.PASSWORD field as an NVARCHAR(255). Using a 1024 bit RSA key the encrypted passwords are 174 characters long. The same password encrypted with a 2048 bit RSA key is just over 340 characters. So, we see what the problem is.

Before I go any further, backup your database. This page isn’t going anywhere, go ahead and do it now.

So, a simple SQL command:

You may or may not get a warning if you are using SSMS (depending on your version I believe) about operations that cause a table to be dropped. The reason you may see this error (even if you don’t see the error it still operates this way) is that in order to change the column’s data type the table is dropped and rebuilt. If you are doing this on a very large table it can take some time to complete, during which time the table is unavailable. At most, I would assume that the hosts table will have a couple hundred rows (the number of rows is the same as the number of ESX hosts you have in vCenter). Even with a few thousand rows, this operation will only take a few seconds to complete.

If you want to be safe, turn off the vCenter service prior to the operation, but I don’t think it’s necessary.

Of course, doing this will probably cause VMware to laugh at you if you open a support issue about the database being broken. Additionally, should you update vCenter 2.5 (say from update 4 to update 5), the change could be undone.

Once you’ve made the change, go about the normal way of replacing vCenter’s certs with your own, and when you reconnect the hosts you shouldn’t get any “string too long” errors.

I haven’t tried replacing the certs for my vCenter 4.0 instance with 2048 bit, so I don’t know if this is still needed, but from reading the documentation it seems that it shouldn’t be.

PowerCLI: Speed boost, Find VM Snapshots by Name.

I use NetApp SnapManager for Virtual Infrastructure(SMVI) to back up 95% of my environment. SMVI isn’t perfect, and it’s kinda a pain, but it gives me the ability to back up 100-250 vm’s in less than 10min! Well the actual NetApp snapshot takes seconds. The rest of the time is spent waiting on ESX snapshots. The only real downside here is from time to time SMVI will fail to delete the ESX snapshots. I have been using PowerCLI to find and delete these snapshot, but the cmdlets are just too slow, for what I’m trying to do. I’ll post my Finalized SMVI cleanup script later. Until then, I give you finding snapshots really fast!

Read morePowerCLI: Speed boost, Find VM Snapshots by Name.

PowerCLI: VM displayName -ne Name

If you were lucky enough to get a hold of @jasonboche vCalendar then this morning you were greeted with a tip regarding the DisplayName of a VM not matching it’s actual path.   As someone who has ran rm -rf  on more than one running vm because of this very issue.**  I will personally attest to why you should care about this!  Jason includes a link to Dominic’s solution in the calendar, but I’m not crazy about perl. So I wrote my own, PowerCLI find any VM with a naming issue.

So what does that do? Well for speed we start with Get-View, and only retrieve the Config property for each VM. Then we pass that collection through Where-Object. This is where PowerShell earns it’s name.

So -match will perform a regular expression comparison operation, So here if the VmPathName looks anything like vm/vm.vmx -match will return $true, and that object will be passed down the pipeline. That however is only part of the story. When PowerShell matches something with a regex it stores the results in $Matches. This means that we can easily look that data up later. In addition I’m naming my matches. if you look in my regex you’ll notice (?<folder>.+?). The ?<folder> will apply the parameter name “folder” to anything that is matched inside the parenthesis, How cool is that!

From there it becomes real easy to see what I’m doing. I compare the displayName with both the Folder name, and the name of the vmx. Then I pass anything that didn’t match to the Get-VIObjectByView Cmdlet.

So how did it do? Well I just scanned 600 VM’s in less than 20 sec, and found two problem children. There you have it, my PowerCLI vCalendar solution for September 10th, 2009! I’m really looking forward to the rest of this calendar, and all the possible automation goodness.

~Glenn

**Note your not screwed if you delete a running VM, take a deep breath, and DON’T power the VM off!  You’ll need to recreate several files, but you have time.  ESX/vSphere have a lock on everything the VM needs to run.