The cons and cons of testing installations with WiX Lux Extension

On my previous post I presented the challenges of testing installations. I've briefly discussed my reluctance with Lux - the only existing framework for unit-testing installations.

In this post I'll show a simple replacement for Lux that only uses standard WiX markup.
First lets summarize the operational concept of Lux tests:

• Immediate custom actions set properties
• Deferred custom actions parse their CustomActionData properties and modify the system accordingly
• Lux tests properties alone, leaving deferred custom actions for other testing methods.

Which reveal the main problem with Lux - its limited scope: It only tests property values. It doesn't check deferred actions' results, nor does it check table values. It doesn't even check that an installation passes successfully - same goes for removal, upgrade, or any other flow.

An example to a basic Lux test - taken from the extension's manual - looks like this:

<Fragment>
  <lux:UnitTest CustomAction="TestCustomActionSimple" 
           Property="SIMPLE" 
           Value="[INSTALLFOLDER]" 
           Operator="equal" />
</Fragment>

This markup tests that a property named SIMPLE equals INSTALLFOLDER property.

One can replace it with this markup, which makes no use of Lux:

<Fragment>

<?ifdef UNIT_TEST ?> 

  <UI>
    <Error Id="10000">UnitTest failed</Error>
  </UI>

  <CustomAction Id="TestCustomActionSimple" 
           Error="10000" 
           Execute="immediate"  
           Return="check" />

  <CustomActionRef Id="WixExitEarlyWithSuccess" /> 

  <InstallExecuteSequence>
      
      <!-- Error if conditions show failure. -->
      <Custom Action="TestCustomActionSimple" Before="InstallFinalize">
        Not (SIMPLE=INSTALLFOLDER)
      </Custom>
      
      <!-- Terminate Successfully if the test passes -->
      <Custom Action="WixExitEarlyWithSuccess" After="TestCustomActionSimple" />
    </InstallExecuteSequence>

<?endif?> 

</Fragment>

This markup defines an error custom action that will be executed if SIMPLE doesn't equal INSTALLFOLDER. If this condition fails (meaning SIMPLE equals INSTALLFOLDER) then the installation will terminate successfully by executing WixExitEarlyWithSuccess.

Lux defines few more test operators, these however are just shims for Windows Installer conditional operators.

Another major drawback of Lux is in its replacement of Product element: Lux replaces the Product element, ignoring anything within it. That means you have to re-organize your markup to be able to test it.
With the alternative markup I presented you don't need to change or re-organize any markup for unit tests to run.

To summarize Lux pitfalls:

• Limited scope - Lux only tests property values
• Lux doesn't check any installation flows
• Product element is ignored altogether.

On my next post I'll present what I think a Windows Installer unit-testing framework should be able to do.

Challenges of Unit-Testing Installations

Every package author has reached the step where a package draft is built and tests are required to find and fix errors.
In many cases testing resolves to eye-check that files reside in the expected folders, and delegated on to QA team for product functional tests.

While most technologies and programming languages are packed with unit-testing frameworks, Windows Installer has none worthy that I encountered. WiX provide a very slim framework called Lux on which I will comment on my next post; however it offers little - if any - capabilities.

Indeed testing an installation package is difficult - files and folders being the easy part. Actually, all standard actions are the easy part as they're already tried and true.
Custom actions are where the problems begin: The basic idea of unit testing is that you can separate small units and test them out of context and independently to the rest of the code.
With installations this isn't true: unless executed by Windows Installer you can't get  valuable information and stimulation, such as built-in properties and execution privileges.

You could design your code in a way that separates input parsing from execution, and test the executing code alone. But even that can be inaccurate.
Examine for example a custom action that kills a process: You may code an immediate custom action to set CustomActionData to the name or ID of the process to kill, and a deferred custom action to kill it.
To test it you will could run the part of that kills the process. Test it with elevated privileges and the test passes; test it as part of an installation and the test fails - that's because Windows Installer - though running with LocalSystem account - runs without SeDebugPrivilege rights, thus can't kill a process.

It seems that installing the product is the only way to test it - same goes for upgrade, uninstall and rollback for each scenario.

In following posts I will expand my reluctance with Lux extension, and suggest a unit-testing framework for installations.