Notes on the act of developing with external repositories (i.e. submodules) in git and pester

PracticalSubmodulesWithGitAndPester.txt

Notes on the act of developing with external repositories (i.e. submodules) in git and pester

*To the reader:* If you are here looking for expert advice, please leave as swiftly as possible. No such advice is present here. This is purely a set of reminders for the physical(?) act of how I handle working with git submodules and pester tests.

This assumes you are working in PowerShell. Version 5, if it matters, with git 2.10 (or later? It was what I had when I wrote this) and pester.

*Downloading and using a repo with submodules* - When I was getting started with this, I NEVER developed submodules from the main module. I schlepped over to the submodule, branch, develop, merge back to master, and then come back here and update the submodule, and I still think that's a good policy for anything other than mass hotfixes.

Clone the repo as normal. Here, the folder PesterNewFixtureModule will be created in the current folder:
	git clone https://github.com/sdoubleday/PesterNewFixtureModule.git PesterNewFixtureModule
Set location into that folder:
	cd .\PesterNewFixtureModule
And run bootstrap the submodules (setup.ps1 or bootstrap.ps1 in my modules will do that):
	git submodule update --init --recursive
That command seems to ignore any submodule that has been populated.


*New Repo* - First, create an empty one in GitHub. I had trouble if I put ANYTHING AT ALL in the repo before my initial push from my local repo, so do that push, THEN come back add a license.

Init the repo on the client to always use the github email and github username. I use my git-init module, and set up my user.name and user.email (GO TO THE USER PROFILE, EMAILS SECTION, TO FIND THE GITHUB EMAIL ADDRESS!).

	git-init -userEmail [email protected] -userName blah -Remote origin -RemoteURL https://github.com/sdoubleday/PesterNewFixtureModule.git

Consider a small readme:
	new-item -ItemType File -Name Readme.md -Value "Imports the SMO assemblies."
	git add . ; git commit -m "Readme commit"; 

Then run this:
	git push --set-upstream origin master

NOW go back and add the license on GitHub.
Then:
	git pull

And now add code.

*Add Submodule* - Run this to add and populate a submodule (do this in the main folder of your repo. I'm assuming a pretty flat solution):
	git submodule add https://github.com/sdoubleday/EncodingHelper.git EncodingHelper
Reference it like this:
	.\EncodingHelper\EncodingHelper.psd1

*Update Submodule - large code changes* - Pattern for this is:
cd to another (often parent, but not initialized into git) directory
, clone the submodule's actual repo
, cd into clone
, run
	.\setup.ps1
, update code
, run tests
, push changes
, back to parent directory
, reclone updated submodule's actual repo
, cd into reclone
, run 
	.\setup.ps1
, run tests to be damn sure everything works
, cd back to parent directory
, delete clone and reclone (ok, I typically actually call them <name of repo> and clone<name of repo>)
, cd to main repo I was originally working on
, run submodule update script (
	.\update.ps1 -force
, i.e.
	git submodule update --recursive --remote
)
, run main module's tests
, declare victory
, pour self a drink.

*Update submodule - mass hotfixes*
But it turns out you CAN actually treat a submodule like its own git repo.
CD into the submodule repo
, make a change
, commit it
, PUSH it (this is important, because your parent repo now thinks it is pointing to the new commit hash, and if that never makes it to the remote, then you have a problem)
, cd back to the parent repo
, rinse and repeat with other submodules if needed
, commit and push parent repo.
Hint: (modified content) is bad, (new commits) is good when you are looking at git status output in the parent repo.

*Mass updating a bunch of freshly cloned repos to use a new version of a submodule* - if you updated a dependency of many of your repos, you could do something like this, but don't work with multiple levels of nesting at once (i.e., resolve all the updates for a module before doing this to a module that submodules the first module). Protip: don't run this INSIDE a repo, run it in the parent directory, or you will crash git. Did you know you can crash git? Well, you can.
	ls -directory | % {push-location; cd $_.FullName; . ".\bootstrap.ps1"; . ".\update.ps1" -force; git add .; git commit -m "Updating submodules for REASONS."; git push; Pop-Location}

*Undo .\Update.ps1 -Force - just then run:
	.\bootstrap.ps1

*Updating a submodule that has had submodules added to it - after you run .\update -force, the submodule's own added submodules will be empty. To fix that, cd into the submodule to which you added submodules (the parent):
	cd .\ParentSubmoduleInQuestion
	.\setup.ps1

*Only updating SOME submodules - This may be pertinent if you commit changes to a module that you submodule many places, then update not the things that submodules that module but a PARENT of those (so, more than one layer away). 
.\Update -Force -Recursive will cause all of those submodules to get the new version. In a perfect world, you would already have wandered around updating all those intervening modules, but you didn't, because REASONS. 
The affected modules will all be marked (modified content) but NOT (new commits). cd into each of those and run:
	.\bootstrap.ps1
That should set everything back the way it should be, and the module will no longer have modified content.

*Run Pester Tests without running all submodule tests* - from your main repo folder:
	powershell -noprofile {Invoke-Pester (ls *tests.ps1 ) }
Or if you are having problems with contaminating your tests with modules in your $env:PSModulePath (importing these cover the cases I have found thusfar for typically-used built-in cmdlets: "Microsoft.Powershell*","CimCmdlets"):
	powershell -noprofile {get-module -listavailable "Microsoft.Powershell*","CimCmdlets" | import-module; $PSModuleAutoloadingPreference = "none"; import-module Pester; invoke-pester (ls *tests.ps1 )}
Here's that again in function form:
	function pesterfoo {    powershell -noprofile {get-module -listavailable "Microsoft.Powershell*","CimCmdlets" | import-module; $PSModuleAutoloadingPreference = "none"; import-module Pester; invoke-pester (ls *tests.ps1 )}  }
	
And if you have too many tests and they make your eyes bleed, this will summarize them by describe block (remove Context from the Group-Object if you REALLY have too many tests):
	powershell -noprofile {get-module -listavailable "Microsoft.Powershell*","CimCmdlets" | import-module; $PSModuleAutoloadingPreference = "none"; import-module Pester; $( invoke-pester (ls *tests.ps1 ) -PassThru -Quiet ).testresult | Group-Object -Property Describe,Context, Passed | select Count, Name   }
	
Or this one (it wrecks the test order, but gives you back the execution times and sorts failures to the top):	
	powershell -noprofile {get-module -listavailable "Microsoft.Powershell*","CimCmdlets" | import-module; $PSModuleAutoloadingPreference = "none"; import-module Pester; $($( invoke-pester (ls *tests.ps1 ) -PassThru -Quiet ).testresult | Group-Object -Property Passed,Describe,Context -AsHashTable -AsString ).GetEnumerator() | % {$_ | Select-Object Name,  @{name='DurationMilliSeconds';Expression={$($_.Value.Time.TotalMilliseconds | Measure-Object -Sum).Sum}}, @{name='Count';Expression={$_.Value.Count} }   }  | Sort-Object Name  } 
	
At which point you then run this, but fill in your problem describe blocks:
	powershell -noprofile {get-module -listavailable "Microsoft.Powershell*","CimCmdlets" | import-module; $PSModuleAutoloadingPreference = "none"; import-module Pester; invoke-pester (ls *tests.ps1 ) -TestName ***DescribeBlockNameHere*** }

*Pester Testing Error Handling* - I use this pattern to get a clean copy of the error message (piping to out-string, for example, adds final carriage return and linefeed characters and can insert others if your string would wrap in the invoking process's window):

	$ErrorMessage = "blah"
	IF "Condition should produce $ErrorMessage"
	$result = try { <#provoke your error here#> } catch {$error[0].Exception.GetBaseException().Message.ToString() }
        $result | SHOULD BE "$ErrorMessage" }

*Redirecting a Submodule to a new Repo* - This is a bit ugly, and has one gigantic caveat, but not that complicated:
Open .gitmodules and update the submodule's url to the new repo:
	notepad .gitmodules
Then run this:
	git submodule sync
And then run this:
	.\bootstrap.ps1
But now the caveat: it pretty much only works for submodules nested one deep, because the parent module keeps config information about all submodules, included nested ones, and it doesn't seem to want to update the nested references.

*Redirecting multiple levels of nested submodules* - I don't know a good way to do this. I suggest you avoid having multiple levels of nesting. 
But if you really want to try it, I've got a script in my Invoke-Git module named DropAndReAddSubmodules.ps1. Take a guess at what it (ostensibly? Hopefully?) does. If you insist on using it, try it in a fresh repo clone.
You will probably need to work one layer of nesting at a time, and it will reorder your .gitmodules file alphabetically (even if it makes no other code changes).

*Patching another repo with submodule related updates* - Write a patch file from the repo with the updated code, which is basically just the output of git diff, to a file. -1 is "the last commit on that branch"
	git format-patch master -1 --stdout -k > ..\PatchFile.txt
Change over to the repo and brach you are patching. Then run a three-way merge:
	get-content ..\..\somewhere\PatchFile.txt | git am -3
If you hit conflicts, run:
	git status
To find the "both modified" files. Open those in an editor, find and update the <<<<<< (version a) ====== (version b) >>>>>> section. Fix that, then run:
	git add . ; git commit ; git am --continue
It will probably tell you there is nothing to commit. Run:
	git am --skip.
And finally, clean up the submodules:
	git submodule sync
	.\bootstrap.ps1

*Checking list of git repos for changes* - For when you want to double check that your many git repos cloned side by side under a directory do not have uncommitted changes
	Get-ChildItem -Directory | ForEach-Object {Push-Location ; Set-Location $_ ; Write-Verbose $_.fullname -Verbose; git status; Pop-Location}

*Changing Git Branches with different submodule commits* - If you change a submodule and run .\update.ps1, then change branches without the intent of merging, you will need to set the actual content of your submodules back to whatever it was on the branch you are on now. Bootstrapping actually does that:
	.\setup.ps1

*Repeated submodule updates can require a full recloning* - If you change enough submodule code, you may wind up with your local repo out of synch with the remote version. You really need to rerun setup, and one way to do that is  to reclone your repo (without running setup), switch to the branch you were working on, and run setup. 

*REMOVE a submodule*
	git rm the_submodule
	remove-item -recurse -force .git/modules/the_submodule