Tuesday, September 30, 2014

Challenges of Unit-Testing Installations

Every package author has reached the step where a package draft is built and tests are required to find and fix errors.
In many cases testing resolves to eye-check that files reside in the expected folders, and delegated on to QA team for product functional tests.

While most technologies and programming languages are packed with unit-testing frameworks, Windows Installer has none worthy that I encountered. WiX provide a very slim framework called Lux on which I will comment on my next post; however it offers little - if any - capabilities.

Indeed testing an installation package is difficult - files and folders being the easy part. Actually, all standard actions are the easy part as they're already tried and true.
Custom actions are where the problems begin: The basic idea of unit testing is that you can separate small units and test them out of context and independently to the rest of the code.
With installations this isn't true: unless executed by Windows Installer you can't get  valuable information and stimulation, such as built-in properties and execution privileges.

You could design your code in a way that separates input parsing from execution, and test the executing code alone. But even that can be inaccurate.
Examine for example a custom action that kills a process: You may code an immediate custom action to set CustomActionData to the name or ID of the process to kill, and a deferred custom action to kill it.
To test it you will could run the part of that kills the process. Test it with elevated privileges and the test passes; test it as part of an installation and the test fails - that's because Windows Installer - though running with LocalSystem account - runs without SeDebugPrivilege rights, thus can't kill a process.

It seems that installing the product is the only way to test it - same goes for upgrade, uninstall and rollback for each scenario.

In following posts I will expand my reluctance with Lux extension, and suggest a unit-testing framework for installations.


  1. Hello,
    The Article on Challenges of Unit-Testing Installations is very informative. It give detail information about it .Thanks for Sharing the information on unit testing. Software Testing Services

  2. This comment has been removed by the author.

  3. One approach I've taken is to write a C# based validator, which gets compiled alongside the *.msi it validates.

    This at least ensures that the input into my C# custom actions passes known error conditions. (But doesn't solve the larger problem of actually running the code in the custom actions to test it)

    One way we've tried this is to do Continuous Deployment, where a build is failed if a deployment is not successfully run on a QA machine. Where we are headed is to using cloud VM's to spin up an environment and execute the installer in a clean environment, and then destroy the VM when done. This is proving to be difficult and will be very time consuming to add to a build.