Combining PowerShell, Bolt and Puppet Tasks - Part 2

In Part 1 of this blog servies we created and ran a PowerShell script on our local computer, and then turned that into a Bolt Task. We then packaged the module so others could use the task. In Part 2 we going to look at how to test the PowerShell script.

Why test tasks?

So the most obvious question is, why would I want to test my Puppet Tasks? When we first start writing Tasks it is true that testing isn't really at the front our minds. However if you stop and look at the actions we took in Part 1, you can start to see we were actually testing our code, it was just a manual process, for example;

When we wanted to add Task metadata we did the following;

Added the metadata information to the update_history.json file
Ran the bolt task show command
Verified that the information output from the command (Step 2) was the same as information we added in the metadata file (Step 1)

In software testing this is known as Arrange, Act, Assert. So we can think of what we did as;

(Arrange) Added the metadata information
(Act) Ran the bolt command
(Assert) The output from the command is the same as the metadata information

So this means we're already doing some kind of testing, but it still doesn't answer the question of "Why should I test?". Testing our tasks means that as we add functionality or change things we can be sure that it still behaves the same way. And by using an automated testing tool (because let's be honest who likes manual testing anyway!) we can run the tests in our module CI pipeline.

What testing tools are out there?

While we could use a testing tool to create a Windows Virtual Machine, in the case of the WSUS Client Module this would be difficult and time consuming to do. So instead we can use Pester which is a testing and mocking framework for PowerShell. In fact it's one of only a few Open Source projects which is shipped in Windows itself!.

Note - this blog post won't go into how to write Pester tests. A quick search will help you, as well as some great talks by Jakub Jares and myself, Glenn Sarti.

Writing testable PowerShell

Source Code Link

Great, so we can use Pester to test our PowerShell task file, but ... there is a problem. In order to test the script we need to import it. We do this by dot sourcing the test script. However this actually runs the script and outputs information.

PS> . .\tasks\update_history.ps1
[
    {
        "ServiceID":  "",
        "Title":  "Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.279.737.0)",
        "UpdateIdentity":  {
                               "RevisionNumber":  200,
                               "UpdateID":  "7cfce973-b755-460c-a1a4-e92512ae2dec"
                           },
        "Categories":  [
                           "Windows Defender"
                       ],
        "Operation":  "Installation",
        "Date":  "2018-10-29 06:55:40Z",
        "ResultCode":  "Succeeded"
    },
    {
...

Also, because the script is written with the logic in the root, instead of in a function, we have no easy way to execute the script in our tests.

In short, the code I wrote may work, but it was not easily testable!

Wrapping the main function

Firstly we need to be able to separate loading the script and running the script. To do this we needed to move all of the logic into it's own function. For example, if the script used to look like;

$Session = New-Object -ComObject "Microsoft.Update.Session"
$Searcher = $Session.CreateUpdateSearcher()
# Returns IUpdateSearcher https://msdn.microsoft.com/en-us/library/windows/desktop/aa386515(v=vs.85).aspx

$historyCount = $Searcher.GetTotalHistoryCount()
if ($historyCount -gt $MaximumUpdates) { $historyCount = $MaximumUpdates }
$Searcher.QueryHistory(0, $historyCount) |
  Where-Object { [String]::IsNullOrEmpty($Title) -or ($_.Title -match $Title) } |
  Where-Object { [String]::IsNullOrEmpty($UpdateID) -or ($_.UpdateIdentity.UpdateID -eq $UpdateID) } |
...

We would wrap all of this in a PowerShell function and then call it;

Function Invoke-ExecuteTask($Detailed, $Title, $UpdateID, $MaximumUpdates) {
  $Searcher = Get-UpdateSessionObject
  # Returns IUpdateSearcher https://msdn.microsoft.com/en-us/library/windows/desktop/aa386515(v=vs.85).aspx

  $historyCount = $Searcher.GetTotalHistoryCount()
  if ($historyCount -gt $MaximumUpdates) { $historyCount = $MaximumUpdates }
  $Result = $Searcher.QueryHistory(0, $historyCount) |
    Where-Object { [String]::IsNullOrEmpty($Title) -or ($_.Title -match $Title) } |
    Where-Object { [String]::IsNullOrEmpty($UpdateID) -or ($_.UpdateIdentity.UpdateID -eq $UpdateID) } |
  ...
}

Invoke-ExecuteTask -Detailed $Detailed -Title $Title -UpdateID $UpdateID -MaximumUpdates $MaximumUpdates

Full Source Code

Notice how the function Invoke-ExecuteTask just wraps around the old logic. It still does the same thing, just in a function.

Note - For those more advanced in PowerShell you may ask why I didn't use Cmdlet Binding in the function header. I could have easily defined this is an advanced function however I did not think it was necessary. The input validation already happens at the top of the script and as this is private function, and no user would be explicitly calling it.

Stopping execution

So now we could call the logic of the script in Pester, but we still had the problem of it actually running the script when we imported it. What we needed was a flag of some kind which could tell the script to execute or not when imported. There are a number of different types of flags; Setting environment variables or registry keys of files on disk. However in PowerShell the simpliest method is to just have a script parameter.

Note - Using a script parameter was appropriate for the WSUS Client module but you may prefer to use something else

At the top of the script we added the NoOperation parameter;

...
  [Parameter(Mandatory = $False)]
  [Switch]$NoOperation
...

We also added a simple if statement at the bottom of the script which conditionally execute the script

if (-Not $NoOperation) { Invoke-ExecuteTask }

This created a switch parameter called NoOperation which would default to false, that is, it would execute the script. By using . .\tasks\update_history.ps1 -NoOperation we could tell the script to not execute and just import the functions for testing.

Note - For those more advanced in PowerShell you may ask why I didn't use the WhatIf parameter instead. The WhatIf parameter is more geared towards a user interaction. While yes it could've been used, all we needed was just simple switch parameter. Also, noop or NoOperation, are common terms for Puppet users.

Writing Pester tests

Now that we could successfully import the PowerShell Task file, it was time to write the tests.

Writing simple tests

The first tests simply tested the enumeration functions. These functions converted the number style codes into their text version; for example an OperationResultCode of 1 means the update is "In Progress"

So first we add the Pester standard PowerShell commands;

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path).Replace(".Tests.", ".")
$helper = Join-Path (Split-Path -Parent $here) 'spec_helper.ps1'
. $helper
$sut = Join-Path -Path $src -ChildPath "tasks/${sut}"

. $sut -NoOperation

These commands;

Calculate the name of the script being tested (also known as the System Under Test or $sut) based on the test file name
Import any shared helper functions (spec_helper.ps1). This blog post didn't add any, but in the future they may be used
Imports the script under test. Note the use of the new -NoOperation parameter

When then test each of the enumeration functions to ensure the the conversions of number to text are what we expect. For example the tests for the Convert-ToServerSelectionString function check the output for the numbers 0 to 3

Writing more tests

So now we had some simple tests written, and passing, we could finish off writing the rest of the tests. Fortunately with testing, we should be describing each of our tests in simple english. I decided that the following tests would be sufficient;

More testing issues

Source Code Link

While writing the test it became apparent that the Invoke-ExecuteTask function still wasn't easily testable. The function created aMicrosoft.Update.Session COM object. This object was then used to query the system for update history. However this meant the testing could only query the existing system, and that we wouldn't be able to see the behaviour if there no updates available, or 1000 updates. What we needed to do was mock the respsonse of the COM object so we could test the function properly.

Fortunately Pester provides a mocking feature, however the function needed to be modified so we could mock the response. So again we wrapped the logic in another function:

Previously we had

Function Invoke-ExecuteTask() {
  $Session = New-Object -ComObject "Microsoft.Update.Session"
  $Searcher = $Session.CreateUpdateSearcher()
  # Returns IUpdateSearcher https://msdn.microsoft.com/en-us/library/windows/desktop/aa386515(v=vs.85).aspx

...

and after wrapping the object creation

Function Get-UpdateSessionObject() {
  $Session = New-Object -ComObject "Microsoft.Update.Session"
  Write-Output $Session.CreateUpdateSearcher()
}

Function Invoke-ExecuteTask($Detailed, $Title, $UpdateID, $MaximumUpdates) {
  $Searcher = Get-UpdateSessionObject
  # Returns IUpdateSearcher https://msdn.microsoft.com/en-us/library/windows/desktop/aa386515(v=vs.85).aspx

...

Now we could mock the response from Get-UpdateSessionObject to simulate any number or kind of updates with the testing helpers New-MockUpdateSession and New-MockUpdate.

For example, the should return empty JSON if no history test mocks an update session with no updates, using the Pester Mock function;

  Mock Get-UpdateSessionObject { New-MockUpdateSession 0 }

Failing tests

Source Code Link

Running the Pester tests showed a failure. The should return a JSON array for a single element was failing;

  Describing Invoke-ExecuteTask
    [+] should return empty JSON if no history 472ms
    [-] should return a JSON array for a single element 162ms
      Expected regular expression '^\[' to match '{
          "Categories":  [
                         ],
          "ServiceID":  "d605c6f0-cdea-4b1e-a225-e643254056d4",
          "UpdateIdentity":  {
                                 "RevisionNumber":  3,
                                 "UpdateID":  "d306e6b6-dd95-46ed-be96-137ecddd8611"
                             },
          "Date":  "2018-11-15 14:30:15Z",
          "ResultCode":  "Succeeded With Errors",
          "Operation":  "Uninstallation",
          "Title":  "Mock Update Title 1724034957"
      }', but it did not match.
      82:     $ResultJSON | Should -Match "^\["
      at <ScriptBlock>, C:\Source\puppetlabs-wsus_client\spec\tasks\update_history.Tests.ps1: line 82
    [+] should not return detailed information when Detailed specified as false 156ms
    [+] should return detailed information when Detailed specified as true 74ms
    [+] should return only the maximum number of updates when specified 73ms
    [+] should return a single update when UpdateID is specified 71ms
    [+] should return a matching updates when Title is specified 73ms

This failure turned out to be a valid. When the bolt task runs it should return a JSON Array, even for a single update. This turned out to be a percularity with PowerShell and piping objects. With a single object in the pipe, the JSON conversion just returns the object, whereas with two or more objects the JSON convertsion returns an array.

In this case the fix was fairly simple. I manually added the opening and closing brackets to the string if there was only one object in the pipe! Running Pester again showed all tests passed!

  Describing Invoke-ExecuteTask
    [+] should return empty JSON if no history 42ms
    [+] should return a JSON array for a single element 60ms
    [+] should not return detailed information when Detailed specified as false 99ms
    [+] should return detailed information when Detailed specified as true 61ms
    [+] should return only the maximum number of updates when specified 68ms
    [+] should return a single update when UpdateID is specified 31ms
    [+] should return a matching updates when Title is specified 32ms

Running tests automatically

Source Code Link #1

Source Code Link #2

Having a suite of tests to run was nice, but we really needed them to be run in a Continuous Integration (CI) pipeline. Fortunately the WSUS_Client module was already setup with an AppVeyor CI pipeline;

I added a small helper script to install Pester if it didn't exist and then actually run Pester
I modified the Rakefile which is used by the PDK and Ruby, to run testing tasks. This change called the helper script I created previously
I modified the AppVeyor configuration file to call the new spec_pester Rake task

Now whenever anyone raised a Pull Request, the pester test suite would be run!

Note - Why did I create a Rake task instead of calling the helper script directly? All the of PuppetLabs modules execute Rake tasks in the AppVeyor configuration file. While I could have hacked the configuration to run the script directly it would cause this module to become a unique configuration which is hard to manage over time.

Wrapping up

We modified the Bolt Task PowerShell script to be easily testable and then wrote a test suite. We then configured our CI tool, AppVeyor, to run the tests for new Pull Requests. Now we can more easily make changes to the task and be confident we don't break the existing behaviour

In Part 3, we'll look at how tasks, Puppet Enterprise and PowerShell can integrate together.

Changes to Part 1 of the blog

In Part 2, we'll look at how to test our newly created PowerShell Task

glennsarti/wsus_blog.md

Combining PowerShell, Bolt and Puppet Tasks - Part 2

Why test tasks?

What testing tools are out there?

Writing testable PowerShell

Wrapping the main function

Stopping execution

Writing Pester tests

Writing simple tests

Writing more tests

More testing issues

Failing tests

Running tests automatically

Wrapping up

RandomNoun7 commented Nov 15, 2018

Uh oh!

clairecadman commented Nov 16, 2018

Uh oh!