https://plugins.jenkins.io/azure-vm-agents
I needed nonstandard software (Packer, Hyper-V, ...) so I had to choose Image Configuration = Advanced, then choose the image manually
az vm image list -p MicrosoftWindowsServer --all -otable
Image I wanted:
Offer Publisher Sku Urn Version
----------------------- ----------------------------- ---------------------------------------------- --------------...
...
WindowsServerSemiAnnual MicrosoftWindowsServer Datacenter-Core-1709-smalldisk
I translated the above to
- offer WindowsServerSemiAnnual
- publisher MicrosoftWindowsServer
- sku Datacenter-Core-1709-smalldisk
- version latest
Used custom init script initscript.ps1
in this same gist
Set label to be called "win", and set that for the node in the pipeline
Scheduled a job, and it creates the VM but deletes it after 20 minutes
Well, something's using a custom script extension. Let's go look at what it's trying to run, and run it again so we can see if it fails.
PS C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0> type .\win-agent0127082024664init.ps1
Set-ExecutionPolicy Unrestricted
$source = "https://github.com/PowerShell/Win32-OpenSSH/releases/download/v0.0.16.0/OpenSSH-Win64.zip"
$destination = "C:\OpenSSH-Win64.zip"
$webClient = New-Object System.Net.WebClient
$webClient.DownloadFile($source, $destination)
$shell_app=new-object -com shell.application
$zip_file = $shell_app.namespace($destination)
$dir='C:\Program Files\OpenSSH-Win64'
mkdir $dir
$destination = $shell_app.namespace('C:\Program Files')
$destination.Copyhere($zip_file.items(), 0x14)
[System.Environment]::SetEnvironmentVariable("PATH", $Env:Path + ";${dir}", "Machine")
Set-Location $dir
.\install-sshd.ps1
.\ssh-keygen.exe -A
.\FixHostFilePermissions.ps1 -Confirm:$false
Start-Service ssh-agent
Start-Service sshd
New-NetFirewallRule -Protocol TCP -LocalPort 22 -Direction Inbound -Action Allow -DisplayName SSH
Set-Service sshd -StartupType Automatic
Set-Service ssh-agent -StartupType Automatic
PS C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0> dir 'C:\Program Files\OpenSSH-Win64\'
PS C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0> new-object -com shell.application
Application Parent
----------- ------
System.__ComObject System.__ComObject
PS C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0> dir
Directory: C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 1/27/2018 8:26 AM 1024 win-agent0127082024664init.ps1
PS C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0> .\win-agent0127082024664init.ps1
Error HRESULT E_FAIL has been returned from a call to a COM component.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:9 char:1
+ $zip_file = $shell_app.namespace($destination)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
mkdir : An item with the specified name C:\Program Files\OpenSSH-Win64 already exists.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:11 char:1
+ mkdir $dir
+ ~~~~~~~~~~
+ CategoryInfo : ResourceExists: (C:\Program Files\OpenSSH-Win64:String) [New-Item], IOException
+ FullyQualifiedErrorId : DirectoryExist,Microsoft.PowerShell.Commands.NewItemCommand
You cannot call a method on a null-valued expression.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:13 char:1
+ $destination.Copyhere($zip_file.items(), 0x14)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
.\install-sshd.ps1 : The term '.\install-sshd.ps1' is not recognized as the name of a cmdlet, function, script file,
or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and
try again.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:18 char:1
+ .\install-sshd.ps1
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.\install-sshd.ps1:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
.\ssh-keygen.exe : The term '.\ssh-keygen.exe' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:19 char:1
+ .\ssh-keygen.exe -A
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.\ssh-keygen.exe:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
.\FixHostFilePermissions.ps1 : The term '.\FixHostFilePermissions.ps1' is not recognized as the name of a cmdlet,
function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:20 char:1
+ .\FixHostFilePermissions.ps1 -Confirm:$false
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.\FixHostFilePermissions.ps1:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Start-Service : Cannot find any service with service name 'ssh-agent'.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:22 char:1
+ Start-Service ssh-agent
+ ~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (ssh-agent:String) [Start-Service], ServiceCommandException
+ FullyQualifiedErrorId : NoServiceFoundForGivenName,Microsoft.PowerShell.Commands.StartServiceCommand
Start-Service : Cannot find any service with service name 'sshd'.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:23 char:1
+ Start-Service sshd
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (sshd:String) [Start-Service], ServiceCommandException
+ FullyQualifiedErrorId : NoServiceFoundForGivenName,Microsoft.PowerShell.Commands.StartServiceCommand
Name : {f354258b-2ad7-439b-851a-78f13f469ace}
DisplayName : SSH
Description :
DisplayGroup :
Group :
Enabled : True
Profile : Any
Platform : {}
Direction : Inbound
Action : Allow
EdgeTraversalPolicy : Block
LooseSourceMapping : False
LocalOnlyMapping : False
Owner :
PrimaryStatus : OK
Status : The rule was parsed successfully from the store. (65536)
EnforcementStatus : NotApplicable
PolicyStoreSource : PersistentStore
PolicyStoreSourceType : Local
Set-Service : Service sshd was not found on computer '.'.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:26 char:1
+ Set-Service sshd -StartupType Automatic
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.:String) [Set-Service], InvalidOperationException
+ FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.SetServiceCommand
Set-Service : Service ssh-agent was not found on computer '.'.
At C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.9\Downloads\0\win-agent0127082024664init.ps1:27 char:1
+ Set-Service ssh-agent -StartupType Automatic
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.:String) [Set-Service], InvalidOperationException
+ FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.SetServiceCommand
Hmm, COM failure. well, this looks suspicious
$shell_app=new-object -com shell.application
$zip_file = $shell_app.namespace($destination)
$dir='C:\Program Files\OpenSSH-Win64'
mkdir $dir
$destination = $shell_app.namespace('C:\Program Files')
$destination.Copyhere($zip_file.items(), 0x14)
Yeah, that's now going to work on an OS without a shell. Why didn't they use Expand-Archive
? Let's give it a go
PS C:\Program Files\OpenSSH-Win64> Expand-Archive C:\OpenSSH-Win64.zip
PS C:\Program Files\OpenSSH-Win64> dir
Directory: C:\Program Files\OpenSSH-Win64
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 1/27/2018 8:34 AM OpenSSH-Win64
PS C:\Program Files\OpenSSH-Win64> cd .\OpenSSH-Win64\
PS C:\Program Files\OpenSSH-Win64\OpenSSH-Win64> dir
Directory: C:\Program Files\OpenSSH-Win64\OpenSSH-Win64
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 1/27/2018 8:34 AM OpenSSH-Win64
PS C:\Program Files\OpenSSH-Win64\OpenSSH-Win64> cd .\OpenSSH-Win64\
PS C:\Program Files\OpenSSH-Win64\OpenSSH-Win64\OpenSSH-Win64> dir
Directory: C:\Program Files\OpenSSH-Win64\OpenSSH-Win64\OpenSSH-Win64
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 6/12/2017 6:00 PM 2276 FixHostFilePermissions.ps1
-a---- 6/12/2017 6:00 PM 609 FixUserFilePermissions.ps1
-a---- 5/24/2017 12:37 PM 4178 install-sshd.ps1
-a---- 5/12/2017 10:24 AM 1462907 libcrypto-41.dll
-a---- 6/12/2017 6:00 PM 1053 OpenSSHUtils.psd1
-a---- 6/13/2017 2:38 PM 23402 OpenSSHUtils.psm1
-a---- 6/15/2017 5:05 PM 317440 scp.exe
-a---- 6/15/2017 5:05 PM 334848 sftp-server.exe
-a---- 6/15/2017 5:05 PM 384512 sftp.exe
-a---- 6/15/2017 5:05 PM 515072 ssh-add.exe
-a---- 6/15/2017 5:05 PM 607232 ssh-agent.exe
-a---- 6/15/2017 5:05 PM 632832 ssh-keygen.exe
-a---- 6/15/2017 5:05 PM 569344 ssh-keyscan.exe
-a---- 6/15/2017 5:05 PM 133632 ssh-shellhost.exe
-a---- 6/15/2017 5:05 PM 856576 ssh.exe
-a---- 6/15/2017 5:09 PM 899584 sshd.exe
-a---- 4/24/2017 1:37 PM 3520 sshd_config
-a---- 11/8/2016 10:59 AM 567 uninstall-sshd.ps1
PS C:\Program Files\OpenSSH-Win64\OpenSSH-Win64\OpenSSH-Win64> move * ..\..
PS C:\Program Files\OpenSSH-Win64\OpenSSH-Win64\OpenSSH-Win64> cd ..
PS C:\Program Files\OpenSSH-Win64\OpenSSH-Win64> cd ..
PS C:\Program Files\OpenSSH-Win64> dir
Directory: C:\Program Files\OpenSSH-Win64
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 1/27/2018 8:34 AM OpenSSH-Win64
-a---- 6/12/2017 6:00 PM 2276 FixHostFilePermissions.ps1
-a---- 6/12/2017 6:00 PM 609 FixUserFilePermissions.ps1
-a---- 5/24/2017 12:37 PM 4178 install-sshd.ps1
-a---- 5/12/2017 10:24 AM 1462907 libcrypto-41.dll
-a---- 6/12/2017 6:00 PM 1053 OpenSSHUtils.psd1
-a---- 6/13/2017 2:38 PM 23402 OpenSSHUtils.psm1
-a---- 6/15/2017 5:05 PM 317440 scp.exe
-a---- 6/15/2017 5:05 PM 334848 sftp-server.exe
-a---- 6/15/2017 5:05 PM 384512 sftp.exe
-a---- 6/15/2017 5:05 PM 515072 ssh-add.exe
-a---- 6/15/2017 5:05 PM 607232 ssh-agent.exe
-a---- 6/15/2017 5:05 PM 632832 ssh-keygen.exe
-a---- 6/15/2017 5:05 PM 569344 ssh-keyscan.exe
-a---- 6/15/2017 5:05 PM 133632 ssh-shellhost.exe
-a---- 6/15/2017 5:05 PM 856576 ssh.exe
-a---- 6/15/2017 5:09 PM 899584 sshd.exe
-a---- 4/24/2017 1:37 PM 3520 sshd_config
-a---- 11/8/2016 10:59 AM 567 uninstall-sshd.ps1
PS C:\Program Files\OpenSSH-Win64> .\install-sshd.ps1
[SC] SetServiceObjectSecurity SUCCESS
[SC] ChangeServiceConfig SUCCESS
[SC] ChangeServiceConfig2 SUCCESS
sshd and ssh-agent services successfully installed
Well, that's didn't do it. the Azure Agent VM plugin is still not connecting. Checking out the VM again...
PS C:\Program Files\OpenSSH-Win64> net start sshd
The sshd service is starting.
The sshd service could not be started.
A system error has occurred.
System error 1067 has occurred.
The process terminated unexpectedly.
Bug filed https://issues.jenkins-ci.org/browse/JENKINS-49212
Since server core is a dead end with the current plugin, go back to the Windows Server 2016 datacenter w/ UI
- publisher WindowsServer
- offer MicrosoftWindowsServer
- sku 2016-Datacenter
It seems to deploy, then start running my custom init script. I ran into some issues downloading Packer with System.Net.WebClient.downloadFile(), so I switched to Start-BitsTransfer
which works.
I'm not 100% sure if persisting $ENV:Path
and rebooting is working yet. When an agent is brought up, I hit errors like this:
...
> git rev-parse --verify HEAD # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
GitHub has been notified of this commit’s build result
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2675)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3150)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:859)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:355)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to win-agenedd830
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
at hudson.remoting.Request.call(Request.java:192)
at hudson.remoting.Channel.call(Channel.java:907)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
at com.sun.proxy.$Proxy252.clean(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:450)
at hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:30)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1238)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:329)
at hudson.remoting.Channel.terminate(Channel.java:992)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:96)
Finished: FAILURE
Going to Nodes > (name) > Status and clicking launch agent. forgot exact button. Then it gets further:
Checking out Revision 65b26f4b7f2a660f5adb4468a8d7057e7bf697a3 (refs/remotes/origin/hyperv-insider-gen2)
> git config core.sparsecheckout # timeout=10
> git checkout -f 65b26f4b7f2a660f5adb4468a8d7057e7bf697a3
> git branch -a -v --no-abbrev # timeout=10
> git checkout -b hyperv-insider-gen2 65b26f4b7f2a660f5adb4468a8d7057e7bf697a3
Commit message: "Fixing image selection for insider"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (packer build)
[Pipeline] sh
[C:\Users\agentadmin\workspace\hyperv-insider-gen2-jenkins-F2UCPYHYTG47FPKWNE5GR4MDA5HOKEXTTNH2C5UBELPVREGETLOA] Running shell script
+ packer.exe build --only=hyperv-iso windows_server_insider_docker.json
C:/Users/agentadmin/workspace/hyperv-insider-gen2-jenkins-F2UCPYHYTG47FPKWNE5GR4MDA5HOKEXTTNH2C5UBELPVREGETLOA@tmp/durable-8407bda6/script.sh: line 2: packer.exe: command not found
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
GitHub has been notified of this commit’s build result
ERROR: script returned exit code 127
Finished: FAILURE
Why is this failing? I went back to http://127.0.0.1:8080/computer/win-agenf335b0/log and checked the logs.
Start-BitsTransfer won't work over SSH, and Jenkins connects via SSH.
Start-BitsTransfer : The operation being requested was not performed because the user has not
logged on to the network. The specified service does not exist. (Exception from HRESULT:
0x800704DD)
At C:\init.ps1:38 char:1
+ Start-BitsTransfer $source -Destination $destination
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-BitsTransfer], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException,Microsoft.BackgroundInt
elligentTransfer.Management.NewBitsTransferCommand
Tried Invoke-WebRequest, also failing:
Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.
At C:\init.ps1:39 char:1
+ Invoke-WebRequest -UseBasicParsing $source -OutFile $destination
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invok
e-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeW
ebRequestCommand