I solved a problem with Powershell.

We’ve been having some strange issues at work with our email archive solution. The “fix” from the vendor, was to install some software on all of our backend exchange servers…. Yeah that’s how I felt about it as well. Being the kind of change that you must test it was time to set up a lab. I grabbed my white board and wrote down what was needed.

1 Domain Controller
3 Exchange Servers
2 SQL servers
2 WIN2K (email archive)

Without really even thinking about it I opened PowerShell, and went about getting this done. Four commands and 1hr later we were installing exchange!

[28]PS>Get-VirtualSwitch -VMHost (Get-VMHost esx15) -Name DEV04 | New-VirtualPortGroup -name "" -VLanId 900

[29]PS>$folder = New-Folder -name DEV04 -Location (Get-Folder vm)

[36]PS>1..8 | % { New-VM -Name “EMAILDEV$($_)” -ResourcePool (get-ResourcePool ADMIN_DEV) -Location $folder -Template (get-template WIN2K3_ENT_x86) -OSCustomizationSpec (Get-OSCustomizationSpec WIN2K3_ENT_x86)
-datastore (get-datastore vmdata0) -VMHost (get-vmhost esx15) -RunAsync }

[263]PS>Get-Folder DEV04 | get-vm | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName “”

A couple of things to note it took 4 or 5 -whatif attempts to get the new-VM cmd right, and the hour was waiting for the vm’s to deploy.  I know that there is nothing new here, but I often see too many “case studies” with powershell… they overcomplicate things.

yesterday I solved a problem with Powershell.  I didn’t write a script.  I didn’t need v2 remoting.   Just four very simple yet incredibly powerful commands…    Give it a try some time just open posh, and TRY to solve a problem… I’ll admit that it is daunting at first, but after a while the reward is ten fold.


P.S. Did I mention that the VI Toolkit is friggin amazing!

Sometimes I just need Linux

I confess… Sometimes I just need Linux. I’ve tried everything from live disks to running an “Administrative” VM.  The main problem with all of them is the break. What would take 5 min if I could do it from within windows, takes 30min and destroys my thought process along the way.   There are tools that I can run from within windows.  Unfortunately most of them either suck or cost too much.  I don’t have room for another PC/thin client at my desk, and I’m NOT giving up my Vista rig… What to do?

Enter DSL (Damn Small Linux) I stumbled upon this little treat on accident a couple weeks back. They have many versions available and I’ve played with several, but the QEMU VM is simply brilliant.

Not only is It easily configured, but the dang thing just works.  Now when I need to modify a config file on a NetApp, or fix a VMX. I fire up DSL, it launches as a windowed app (on my vista x64 rig!).  90 seconds later I’m logged in and continue working, NO break in work flow!

Couple disclaimers: I still hate it, and think it is beyond legacy, but there is no fighting it anymore. Some form of UNIX will always be in the data center.  So for now I’m using DSL to fill that gap…  Got a better solution?


Powershell OnTAP

It just keeps getting better!  I have personally mentioned the need for a PowerShell SDK to NetApp several times.  The funny thing is both their engineers, and professional services guys never understood why I would want such a thing.  This is a weird paradox with POSIX guys.  They seem to think that connecting via ssh and writing bash scripts is all that is needed.  Sometimes Perl is invoked to provide a higher level of abstraction.  I guess their view is only Linux/Unix guys are going to want to automate this stuff.  It’s the classic windows stereotype; Windows guys only know how to drive the GUI. 

We’ll adweigert saw things differently, and has started converting the Perl SDK into a PowerShell SDK.  At this point it is still very early, but he has already added quite a bit of functionality.  The SDK is currently in the form of a single ps1 script.  Which is actual rather cool, because while I’m not a dev.  I am an OnTAP Administrator, and PowerShell scripter. I can actually contribute to this!

I’m very excited to get to work Monday and give this baby a whirl!  Check it out http://www.codeplex.com/PowerShellOnTAP


VI admins are diggin’ PowerShell

Short of walking around the Venetian shouting “anyone else like PowerShell”.  I did everything i could to evangelize the product, and find likeminded individuals.  The giant posh sticker on my laptop didn’t hurt in that regard.  The one thing I did find when talking to these guys is they are starved for content.  Sadly most of them knew only of the vmware community forums.  Don’t get me wrong Lucd is amazing, but that is just a small piece of the community.  I was very disappointed that neither Carter nor Dimitri Dmitry mentioned any of the resources out there for people. 

I was however blown away by VMware and their commitment to the product.  Quest manned a booth in VMware’s exhibit.  For three days I never saw that booth empty.  They were also handing out copies of powergui, some stickers, and a card… Hal you’ll be pleased to know that I saw many handed out, but not one on the floor.  After the PowerShell session I was talking to a couple VI admins, and one of them recommended your book to me!


(Update:  Don’t read into my post that I didn’t enjoy/value Carters presentation.  I thought it was easily top five for me, and he made the case why VI admins need to learn PowerShell.  My point was that once they start that journey.  The more tools you provide the likeier they are to succeed!  Oh yeah and sorry I mispelled your name Dmitry!)



VMworld is over and all I have to say is Amazing!  I’m still decoding my notes, and writing my post trip report for work.  Earlier when I declared to post updates during the show… I had no idea how busy I would be.  I brought my laptop the first two days, and never even booted the thing.  I filled a 200 page notebook, and my live scribe ran out of room on the last day.  All and all great trip, and should be able to provide enough value that my boss will send me back (please!)

On to the good stuff… Andrew and I went through the solutions exchange only a handful of times.  In that small time frame (maybe 6 hours) we amassed a rather impressive pile of swag.


  • 93 Pens
  • 6 golf balls
  • 7 stress balls
  • 15 1GB thumb drives
  • 4 water bottles
  • 2 USB speakers
  • 3 hats
  • 10 notebooks
  • 4 Laptop bags
  • 4 stuffed animals
  • 15 t-shirts
  • 2 laser pointers
  • 5 flashlights
  • 1 Mac book Air
  • 1 Mac super drive

On the last day of the conference after all of our sessions were done.  We did one last lap through the solutions exchange.  One particular vendor ‘vmSight’ had all these bobble head comic book figures.  I stopped to see if I could talk them out of one.  As is the tradition at these events, stopping meant getting scanned.  Mark from vmSight told me no to the toy, but did explain their product.  (Sounds very cool, I especially like being able to put hard performance numbers on a VDI implementation.) Back to the point, getting scanned entered us in a contest, and while we there Andrew won a Mac book air!  Which he is currently trying to free from osX.  Anyone know a way to boot Linux off a Mac book without boot camp?

VMworld 2008

Sunday, Andrew and I depart for five days of geekin it up goodness.  We will try and post updates through the conference.  I figure we’re going to take notes anyways what’s the difference?  If you want to follow us on twitter we’re http://twitter.com/glnsize, and http://twitter.com/acsulli.

I plan on spending most of my time talking to fellow admins.  I need to gauge what we’re doing in respect to the industry… After the small talk, you’ll probably find me in anything that has the words; PowerShell, Tech Preview, NetApp, Cisco, Exchange, or Active Directory… oh and Beer!

My schedule:


10:00 -11:00 — EA2372   —  Virtualizing Big Applications” – Performance Considerations” 
11:00 -12:00 — KN Cisco —  Designing the Next Generation Data Center – Cisco and VMware
13:00 -14:00 — AD2764  — Managing VMware with PowerShell
14:30 -15:30 — TA1402   —  Introduction to Storage VMotion
16:00 -17:00 — TA2421  —  DRS Technical Overview and Best Practices


09:30 -11:30 — Lab05    —  VMware Infrastructure  – Security Hardening & Best Practices(VMware VirtualCenter ” VMware ESX & VMware ESXi”)
11:30 -12:30 — PO1861  — VMTN Community Experts
13:00 -14:00 — TA1405  — VMotion Technical Deep Dive
15:00 -16:00 — TA2275  — Tech Preview:  VMware Infrastructure Virtual Networking Future Directions
16:00 -17:00 — TA2377  — Performance Roundtable hosted by the Chief Performance Architect


09:30 -11:30 — Lab09    —  Scripting VMware Infrastructure: Automating, Integrating, and Extending VI
13:00 -14:00 — TA2213 — VMware Infrastructure 3 Storage:  iSCSI Implementation and Best Practices
14:00 -15:00 — VD2345 — Going Deep on Capturing Applications using ThinApp

Backup services state

This morning I had a couple app servers just giving me fits. I turned to powershell to quickly to a diff on the servers. I started to backup the regisry and do a diff there, but decided that I should start somewhere simple. My solution was to “snap” the state of all the services on one that was working. I then restored those setting on one of the trouble nodes, rebooted problem solved.

gwmi win32_service | % { write-output "sc config $($_.name) start= $(($_.startmode).replace('Manual','demand')) " } | out-file restore_service.bat

ITIL, U-TIL, we all scream for…Configuration Management?

Ok, so the title is a little misleading. Configuration Management is a part of ITIL, however I’m not going to talk about ITIL, at least not directly.

As an administrator I’m responsible for multiple systems. Some of these are identical, e.g. Apache servers, MySQL servers, some of them provide unique, stand alone, services. However, they all have some things in common…sshd configuration, log rotation schedules (logrotated), etc.

It’s a PITA to keep up with all of these servers individually. A global change can take quite a bit of time, especially with our ever increasing number of ESX hosts. So, how do I make my job easier, myself more productive, and next year’s raise larger? Automated configuration management.

Read more

Punctuality is Important

Time keeping is especially important for Active Directory and Kerberos. I encountered an error when I was attempting to ssh into one of my AD enabled ESX hosts. The SSH error was “Permission Denied”, however after inspecting the logs (/var/log/messages) I discovered that pam_krb5 was throwing “Clock skew too great” errors.

This was odd to me, as I know every one of the ESX servers has NTP configured. Apparently ntpd died at some point, which caused the clock to begin losing time. Once the time difference between the domain controller and the ESX host exceeded 300 seconds (5 minutes), ESX no longer allowed me to login using AD credentials.

The fix was somewhat easy…reset the clock. Since I was able to login to the console, I did so as root, and executed ntpdate name.of.domain.controller, which forced it to sync the clock with the DC. After that was taken care of (which confirmed that it was ntp that broke), I went back to Virtual Infrastructure Client and reset the NTP settings for the host (it’s on the Configuration tab).

Provisioning server for VM’s

Andrew and I recently reorganized our VI at work.  One of the key changes was the concept of datacenters via function… Without getting to far into it. One of the functions we identified, a stand alone resource pool to deploy all VM’s from.   We’re referring to it as our provisioning cluster.  Basically whenever we get a request for a new VM.  The VM is deployed there and then VMotioned to it’s appropriate resource pool only after everything is verified, and documented.

Well done with the theory.  I started to organize all our templates through VIC, but quickly relied we have a ton of them!  Win2k, Win2k3, Win2k8, RELH4, Solaris 10… all or which have x86/x64 variants for each of our licensed options… Standard, Enterprise, Datacenter… etc.  Did I mention all I did for a month was build templates.  Anyways they where everywhere, and I was not looking forward to this.  Then I read this post From Hal’s blog, and quickly realized that with was something worth scripting.  The version I used at work looked like

Get-Template | get-view | % {$_.MarkAsVirtualMachine((get-cluster "pool1" | Get-ResourcePool | get-view).MoRef, (get-VMHost "ESX1.localdomain.local" | get-view).MoRef); $_.MarkAsTemplate()}

but that kids is not what I would call production ready.  That’s what I love about PowerShell I had a one time task… boom one line!  Took 15 min to find/move all of the templates we had in our env.   As I was added this script to our internal Wiki it occured to me someone could probably build on this the same way I built on Hal’s post.  So here is a slightly more polished version.