Windows Sysprep and VM creation

I've seen a ton of blog posts, reference documents and white papers all instructing you—the virtualization admin—to build "template" VMs in the following fashion:

1. Create a VM and install the OS
2. Install standard applications
3. Set all your configuration settings
4. Patch & Update the OS and applications
5. Sysprep
6. Convert to Template

I'm here to tell you now: stop doing Step 5. Don't sysprep those golden images. At least, don't do it in your template process.

At the very least, using this model means you won't be able to update that template more than 3 times: doing a standard sysprep—without a custom unattended settings file—will "rearm" the software activation only so many times. If you run out of "rearms" you get the joy of rebuilding your golden image.

There is a way around the sysprep limit—see the SkipRearm post for my method—but that still leaves you with a VM template that's going to roll through the remainder of the Sysprep process the first time you turn it on—which you'll be doing every time you want to patch or update the image.

Instead, make Sysprep part of your new VM creation process. With VMware, you can easily convert a VM  back-and-forth from a template to a VM; in fact, for the longest time, I never even converted VMs to templates because there didn't seem to be much value in them: everything you could do to a template, you could do to a VM, while there are things you can do with a VM that you can't do to a template.

Instead, leave your golden image at Step 4; you will be revisiting it every month anyway, right?

Every time you need to spin up a VM from that point forward, you will have a (relatively) recently-patched starting point. In fact, if you're really efficient, you'll run the template VM before creating a new machine from it and patch that machine. Either way, you'll be patching a VM; but if you need to spin up more than one VM, the patching is already complete!

So here's my process:

A) Create your golden image

B) Update your golden image

C) Clone a new VM from the golden image

D) Run Sysprep (with or without SkipRearm and other unattended settings)

E) Repeat steps C-D as needed

F) Repeat step B as needed

Note: I realize there are certain setups that require you to leave a template/golden image at the post-Sysprep shutdown state. In those cases, just make sure you've got a snapshot prior to Sysprep so you can revert to a state before it was ever run.

Remove security warning from Internet-sourced files

Ever been setting up or managing a system and run into a prompt like this:

It’s probably because you grabbed the original executable from an Internet site.

Using a modern browser to grab the file will typically result in a special NTFS stream added to the originally-downloaded file (eg, bginfo.zip) that gets promulgated to the executable you’re trying to run.

This can be a good thing when you're trying out software, but how do you fix it when you know you can trust the file? This sort of thing can be come quite annoying if it's tied to a Startup item like BGInfo.

The best solution is to “unblock” the file you download; that keeps the stream from being added to the extracted file(s). But what if you’ve already extracted them?

Same solution, but you apply it to the executable instead of the download. Right-click on the file to unblock, then select properties. You should see something a bit like this:

Note the [Unblock] button at the bottom. If you click that and save the properties, the NTFS stream metadata is removed from the file, and you won’t get the popup message whenever the app is run.

When I'm retrieving trusted files from my own web servers, I’ve simply gotten into the habit of unblocking files as soon as I download them; if the ZIP or installer file doesn’t have that metadata, the extracted files won’t inherit them.

Also: there’s no way to mass-unblock files; if you select a group of files and choose properties, you don’t get the option to edit the security. If you're downloading a zip file full of executables (like the SysInternals Suite), you definitely want to unblock the ZIP file before extracting it, or you'll have to unblock each executable individually.

Making SkipRearm work for you

So, one of the nice parts about virtualizing a Windows 2003 or XP system (other than the small resource footprint, compared to newer OS versions) was the quick, tidy way of cloning and generalizing them: clone it, run NewSID to give it a new SID and NetBIOS name. Done.

We can't do that anymore with Windows Server 2008 R2 and Windows 7 (nor Server 2008 and Vista): between licensing scheme changes and NewSID going the way of the dodo, there are only two ways to virtualize them: install from 'scratch', or clone/sysprep.

Microsoft has made huge strides in their install platform since it was introduced, and doing "from scratch" installs aren't that bad anymore; but if you've got a system that's set up just so, it's probably a lot more work to rebuild from scratch than to clone & generalize.

But that's what can cause problems: the default behavior of sysprep is to reset the product code and licensing activation state for the cloned machine. In and of itself, that's no great issue, but Microsoft built a hard limit into the number of times a system can be "rearmed" for licensing; if you reach that limit, there's no do-overs. You can't get sysprep to succeed.

There's a way to address this, too: Microsoft also recognized that there might be times when you need to leave the machine's licensing state alone, yet still generalize it. You can find articles around the 'net for the "SkipRearm" component of a sysprep answer file, and it does work. Mostly. If you do it correctly.

That's the point of this post: for every way that exists to do it correctly, there are probably 150 ways to do it incorrectly. I know: I spent several hours over the weekend trying to get it working.

I succeeded, but it wasn't quick. So what follows is the documentation for the method that worked for me...

To make shorter work of this, you'll need several things:

1. Microsoft WAIK (Windows Automated Install Kit). It's an ISO that includes the installer for SIM (System Image Manager). The key item is SIM.
2. Install image (WIM) from the OS you're trying to work with. It can be the base install.wim that comes on the distribution media, or an updated WIM that you used to create your "template" system.
3. A VM with a fully-licensed OS. You'll want to run this VM on a hypervisor that will allow you to take snapshots (Type I or Type II, doesn't make a difference--it's the snapshot facility that we're after to make faster work of this).
4. Text editor (Notepad is fine, but I like the highlighting in SciTE, the Scintilla Text Editor)

Assemble your toys, and take a snapshot of your VM so that you can roll it back to the state that exists prior to "messing" with it.

1. Launch SIM and open your install image:
   
   
2. Create or open an answer file:
   
   
3. Expand the Components folder, right-click the Microsoft-Windows-Security-SPP component that is appropriate for your OS type, and select Add setting to Pass 3 generalize. Note: if you instead select the -SLC component, it will have a SkipRearm setting, but the program notes indicate that the setting has been deprecated. In practice, it means "this won't work on newer OSes."
   
   
   
   Additionally, if you're doing this preparation for a 32-bit OS (the screenshots are for Server 2008 R2, by definition a 64-bit OS), you will need to make sure you've selected the x86_ component, not the amd64_ as I've done in the examples. You will note that the Server 2008 R2 WIM doesn't include that option in the components list, but it is available in the 32-bit Windows 7 WIM.
4. In the settings window, change the value for SkipRearm to 1
   
   
5. Close the Windows image. This will remove any specific association to that image from your answer file.
6. Save your answer file. Exit SIM. Open your answer file in a text editor.
7. Note the details in the XML file entries. Those attributes of the component name are the piece that seem to be missing from all the other postings I've seen for this function. If you don't have them all—including that publicKeyToken attribute—your answer file will not work.
   
   
8. If you're not going to play with SIM and try to add additional functionality to your answer file, copy the contents to a file on your VM.
9. Sysprep can be found in c:\windows\system32\sysprep, which is not in the environment path, so you'll need to open a command shell and go to that directory to invoke it. Invoke it with the following command:
   sysprep /unattend:{answer file you created} /oobe /generalize /reboot
   Assuming your answer file was formatted and read correctly, sysprep will take care of generalizing the VM and rebooting. It will take a couple of reboot passes before it's ready for you to work on it, and the default "out of box experience" dialogs will request your attention; when that's complete, you should see that your VM:
1. is still licensed
2. has a new SID
3. has the same number of "Remaining rearm count" as the source VM
10. When you're through testing, revert your VM to the snapshot, delete the snapshot, then save the answer file to the base image.

Once you have an answer file saved to a base VM, it's trivial to clone, sysprep and be on your way with a minimum of effort.

Resize your Vista/Win7/Win08 desktop icons

If you're a clean desktop type, you've figured out by now that you can resize (in smallish increments) the icons that Microsoft puts on the user desktop (notably, the Recycle Bin, which is the only icon on a fresh desktop—unless the administrator has installed something else as well).

Hold down the CTRL key and scroll the mouse.

So if the default icon size looks a bit like this:

You can scroll and make them tiny (my preference):

And you can also increase their size in case you prefer to go that direction:

Unfortunately, there are times when you can't scroll, or scrolling doesn't get interpreted correctly. Working inside a virtual machine is one of those times; working by remote can be another. Luckily, the Windows API allows a programmer to "simulate" a scroll-wheel in code, bypassing the need for a physical scroll wheel.
The Jamie Morrison over at the ether published a small console program that can do just that: programmatic simulation of scroll-wheel events. See their Knowledgebase article for more information.

Update: In the event that the links above ever go dead, here's the version that was available on 12-June-2012

Windows print driver insanity

Windows print drivers are the bane of many admins and users. These bits of software live in the murky area between the OS kernel and user space, and decisions (and non-decisions) made by their developers can quickly turn a well-running system into complete garbage.

It's my opinion that much of the bad press that Windows gets (compared to Linux or OS X) is the result of poorly coded drivers (and I definitely include video driver in this indictment).

The real head-shaker I'm working on, however, isn't the bad behavior of the driver as much as the peripheral settings and installer that come with the driver.

I ran across a situation where the Lexmark "universal driver" was dropping some entries in the Registry that didn't cause issues in single-user environments, but wrought havoc on multiuser (eg, terminal server) environments.

Specifically, the driver saved a binary value that essentially contained an XML file. On the surface, that would be innocuous, but when the size of the blob is on the order of 500KB—and the same blob is duplicated in several values and keys throughout both the System and User hives—it becomes an issue.

Even when Windows has supposedly fixed the Registry memory restriction and allocation problems, starting in Windows 2003.

Not sure if you have a potential to be bitten by this little gem? Well, if you aren't using Lexmark printers, you should be fine. But if you are, open Regedit and do an exact search for the following value name: GDL

I had a terminal server user with over 50 of these entries in her user profile; when she was logged in, no other users could log on (not enough resources for the registry of the new login). We removed the values, and voila, users could login.

Insanity!

(Of course, I now want away to search for Registry entries by size: anything bigger than 100K needs to be flagged so I can complain to the vendor—even if it's Microsoft—about bloating my Registry.)

SQL Server Failover Cluster -- Add-a-node gotcha

This isn't the sort of thing someone does all the time, so you're bound to be surprised by one thing or another...

We run instances of Microsoft SQL Server 2005 on hardware failover clusters (I'd love to run them all on VMware, but there are a number of reasons why we don't.). We had an occasion to replace the hardware for the nodes, and an associate & I were trying to get this accomplished.

Putting the 2 new nodes into the cluster was smooth as butter. That left me with a 5-node cluster (we were going to ultimately evict the older 3 nodes), and I set forth to add the new nodes to the first SQL instance on my list.

But I kept getting stopped by the "remote task will not start" error. I googled for a solution, and kept coming up with remote access issues: in essence, make sure you aren't using Remote Desktop to connect to the new passive node. I wasn't using RDP on any of the nodes, so I couldn't understand why it kept failing. Just about every combination of reboots and node selection was attempted. I made sure I was logged out from the new nodes. Still, no joy.

In final desperation, I logged out of all the nodes, not just the ones I was adding, then logged back in to the instance's active node and ran the setup from there. Voila! I was able to add a new node to the instance. For whatever strange reason, you cannot be logged into ANY of the nodes other than the one from which you're installing. Period. Not using RDP like they document, but also not using VNC or even the actual system console. NONE.

Shutdown without powering off.

Have you ever needed to shut down your PC, but wanted it to stay powered-on for some reason?

In my case, I have a power outage scheduled at a remote location—the power company is going to kill the power to do utility work for about 4 hours—and I really don't want to drive out to the remote location just to turn the server back on.

What I'd prefer to do is use remote access to put the server in a state that is "safe" for the operating system (i.e., I/O is flushed and volumes are clean dismounted), but it is still "on" so that when the power comes back on after being off for a while, the server will happily reboot.

Mind you: I have this server on a UPS, so its power won't fail until the UPS is fairly well depleted and shuts itself off; it won't power back on until the UPS has recharged enough to keep the server running for 20+ minutes. This will eliminate issues with the power company flipping things on-and-off while doing their work, as well as the very good potential for a surge when they finally get things turned back up.

Of course, this could have all been avoided if I had the BIOS set to power-on the server on whenever the power state switches from OFF to ON. However, there are any number of reasons why it's not currently set this way, and resetting it would require two things: driving to the remote site (which I want to avoid) and scheduling an outage because there's no way to edit this particular BIOS setting via administrative software (too bad, eh?). In some ways, resetting the BIOS is a worse option than what I'm after.

So what's the trick?

1. You have to enable a policy for the system
2. Use software (not the START button) to initiate the shutdown.

One can enable the policy in several ways, but the end result is the following registry key being set:

System Key: HKLM\SOFTWARE\Policies\Microsoft\Windows NT
Value Name: DontPowerOffAfterShutdown
Data Type: REG_DWORD
Value Data: 1 (0 = power off, 1 = stay on)

This policy is only observed when software is used to perform the shutdown (like UPS management software, which we haven't got for our server, as the UPS is an old horse without a modern USB interface. Yeah, I know: it also means that the UPS doesn't properly shutdown the system during unplanned outages, but that's an issue for another day...). If you use the START button/Taskbar to perform a shutdown (which is the typical method for a planned/expected shutdown), it will ignore the setting and power down after shutdown.

Personally, I'll be using "shutdown -s -t5" to shutdown 5 seconds after the command is issued.

Other caveats: this policy only works for Windows OSes after XP SP1, and there are known issues with some Pre-SP1 Windows 2003 servers (see Microsoft KB 819760 for more details on that) that might not result in success.