Guide to troubleshooting basic PowerShell script execution via WinRM

[JeremyN (Jeremy Nelson)]

Essential guide to troubleshooting most basic WinRM / PS-Remoting execution issues.

Harness uses WinRM to perform PowerShell (PS) script executions on remote hosts, which comes with a vast number of implications when it comes to getting the proper configuration in place due to innumerable variables such as permissions, policies, general configuration issues, and even some limitations that we will mention in this document.

Typically, in most situations, when you are trying to execute a PS(PowerShell) script on a remote windows host, if it is an actual connectivity issue between the delegate and the host, or an authentication issue. The authentication issues whether it be NTLM or the less utilized Kerberos, can sometimes be more elusive that one might think, as we typically get a relatively generic error during the capability check phase for the eligible delegates requiring us to dig deeper into delegate logs, a very common example being:

‘No eligible delegate was able to confirm that it has the capability to execute [ hostname:failure here. ]’


So let us propose a quick,basic, and ever evolving guide to practice diagnosing the more basic WinRM-related issues we come across, starting with the most common.

I. Connectivity issues between delegate and remote Windows host.

There can be a broad variety of reasons you might get the above failure while trying to execute PS over WinRM. So, let us go in order with the basics, as, unfortunately, the library in use as our WinRM client only sometimes does well at surfacing specific errors. My first recommendation is to confirm the most apparent and essential item, connectivity to the target host from the delegate, especially if the delegate logs are indicative of a connection timeout or similar. Some of these tests will require CLI access to the delegate in question).

A. Test/Verify connectivity for SSL connection using curl command (pre-installed on most of our delegate images). Simply log in and run:

curl --header “Content-Type: application/soap+xml;charset=UTF-8” --header “WSMANIDENTIFY: unauthenticated” https://[host]:[port]/wsman --data ‘<s:Envelope xmlns:s=“http://www.w3.org/2003/05/soap-envelope” xmlns:wsmid=“http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd”><s:Header/><s:Body>wsmid:Identify/</s:Body></s:Envelope>’ -k

A proper response to the above command if the WinRM service is running and allowing unauthenticated connectivity test to WSMAN default endpoint would be something along the lines of:

<s:Envelope xmlns:s=“http://www.w3.org/2003/05/soap-envelope”><s:Header/><s:Body><wsmid:IdentifyResponse xmlns:wsmid=“http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd”>wsmid:ProtocolVersionhttp://schemas.dmtf.org/wbem/wsman/1/wsman.xsd</wsmid:ProtocolVersion>wsmid:ProductVendorMicrosoft Corporation</wsmid:ProductVendor>wsmid:ProductVersionOS: 0.0.0 SP: 0.0 Stack: 3.0</wsmid:ProductVersion></wsmid:IdentifyResponse></s:Body></s:Envelope>

Of course, replace [host] and [port] with the server Hostname/IP and the port in the curl command, which is almost always 5986 in SSL connections(default), and 5985 without SSL over HTTP.

Also, be sure that if you are using the self-signed Windows-generated certificate for the listener, ensure the ‘Skip Cert Check’ field is enabled under the Advanced options of your WinRM credentials connector(also default).

If you get the response example or something similar to what is in this step for the response, you are good to go, at least in terms of basic connectivity, and the wsman service endpoint being available.

Additionally if you do not, and get a timeout, the problem is in fact connectivity as the host would not be listening on said port, and I would recommend verifying the host is properly configured in relation to WinRM listeners(see section IV), network policies between the host and delegate, Windows firewall, and more.

B. Test/Verify SSL certificate(s) being served using OpenSSL s_client (also pre-installed on most delegate image(s):

OpenSSL client is also pre-installed on most delegate images, the next thing to test would be any potential issues with the SSL handshake, this can be performed with the following command from the delegate:

openssl s_client -showcerts -connect server.hostname:port

As mentioned prior, the SSL port for WinRM default is 5986, you should see output like the following:

This lets us know the SSL listener is configured and serving a certificate, even though, in the above case, it is self-signed given the two verify errors which is why we have the Skip Cert Check option enabled by default in the Harness UI for the WinRM credential connectors.

II. Authentication errors, Invalid Credentials, etc.

Here we have example(s) of common issues run into often related to authentication. Unfortunately, the error(s) you might see in delegate logging can potentially be a handful of failure types, but are commonly related to invalid/wrong credentials or a misconfigured WinRM connector, for example take the following very common exception(a lot of it ommitted):

java.lang.IllegalStateException: Invalid credentials or incompatible authentication schemes
at io.cloudsoft.winrm4j.client.RetryingProxyHandler.checkForRootErrorAuthorizationLoopAndPropagateAnnotated(RetryingProxyHandler.java:94 undefined)
at io.cloudsoft.winrm4j.client.RetryingProxyHandler.invokeWithRetry(RetryingProxyHandler.java:47 undefined)
at io.cloudsoft.winrm4j.client.RetryingProxyHandler.invoke(RetryingProxyHandler.java:31 undefined)…
at java.base/java.lang.Thread.run(Unknown Source) Caused by: javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:183 undefined)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145 undefined)
at com.sun.proxy.$Proxy532.create(Unknown Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at io.cloudsoft.winrm4j.client.RetryingProxyHandler.invokeWithRetry(RetryingProxyHandler.java:44 undefined) … 17 common frames omitted Caused by: java.io.IOException: Authorization loop detected on Conduit “{http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd}WinRmPort.http-conduit” on URL “https://10.53.136.37:5986/wsman” with realm “null”
at org.apache.cxf.transport.http.HTTPConduit.detectAuthorizationLoop(HTTPConduit.java:1949 undefined).processAndShip(SignAndEncryptOutInterceptor.java:148 undefined)
at io.cloudsoft.winrm4j.client.encryption.SignAndEncryptOutInterceptor$EncryptAndSignOutputStream.close(SignAndEncryptOutInterceptor.java:84 undefined)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56 undefined)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671 undefined)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63 undefined)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308 undefined)…more frames ommitted

If the above is seen in the delegate logs, the first, and most simple things in order I recommend doing if you have verified connectivity between the windows host and the delegate in the prior section:

A. Run a simple test of the credentials against the host using the WinRM connector that is failing in the deployment or elsewhere, for example:

B. Manually verify the credentials are in fact correct and authorized.

Now there is a chance the last test can still fail here with the credentials being correct, so utilizing this little PowerShell snippet on a remote Windows host with PowerShell installed may help uncover the root of the problem. So, simply copy/paste the below PS script directly into the terminal, or save to a file and execute whichever method you prefer. The following to see if you have the permissions to open a remote PS session on this host with the credentials defined in Harness WinRM connector (which you will be prompted for on Run):

$options = New-PSSessionOption -SkipCACheck -SkipCNCheck
$cred = Get-Credential
$session = Enter-PSSession -ComputerName [remoteHost] -UseSSL -Credential $cred -SessionOption $options -Authentication Negotiate

Of course, replace [remoteHost] with the Hostname/IP of the host above, and remove -UseSSL if you are using port 5985/HTTP, example in my case:

After you enter the credentials (DOMAIN\user) or just user for local account, you should either be dropped into a remote PS session, or receive an ‘access denied’ or similar message indicating you lack permissions to perform this operation on the remote host with said credentials, bringing us to the next section.

III. Permissions-related issues.

In general, permissions is a deep subject when it comes to Windows/Active Directory(NTLM/Kerberos) authentication, so we will not dive deep here, but cover a couple of things to try:

A. First, is the user account being used to do the remote execution on the remote host either a local Administrator or a member of the Domain Admins group in the case of a domain/Active Directory account?

If not, please follow the instructions outlined in this article to fix the issue, as this usually turns out to be one of if not the most common issues arising with failures related to access:

Configure the WinRM service on a Windows host to allow authentication and remoting from non-administrator user accounts

This how-to article is intended to give a pretty basic method of configuring a Windows Server 2012->2019 host to authenticate via NTLM as user or service accounts within an active directory domain that are not administrator users. The following steps should assist in achieving the desired result(normal user or svc account can connect via winrm), with accompanying examples of any relevant configuration(s). Lets get started: Log into the host we are wanting to enable the WinRM service on for …

B. Create a local Administrator account on the remote host to test with.

Next, I recommend just entirely ruling out permissions-related issues, creating a local Administrator user on the remote host if possible, and testing authentication once more using either the script or the test for the WinRM connector previously mentioned in Harness UI. If this is successful, you have a permissions problem that needs to be resolved, which could be one of the items mentioned in section IV below.

IV. Channel Binding Token Hardening Level and Listener configuration(s)

A couple of other items I wanted to mention, usually edge cases, but there is a setting if you run the following from a command prompt in windows:

winrm get winrm/config

A. Please note that the usual setting for the above ‘CbtHardeningLevel’ is relaxed, and for NTLM is a requirement due to the current limitations of our client library. So, if it is set to ‘Strict’, you will need to change this to ‘Relaxed’ to authenticate properly over SSL/TLS using NTLM. In our testing, when utilizing Kerberos, this can be set to with no issue.

B. Lastly, check that the WinRM listener configuration(s) are correctly configured (eg; have a certificate/thumbprint, are enabled (which would cause a connectivity issue mentioned in the first section). You can to this by running the following from a command prompt or PS:

winrm e winrm/config/listener

The above command will provide a list of all listeners configured for WinRM and show essential items such as app layer transport type, whether they are enabled, the interfaces they are bound to, and much more.

To conclude, this is just scratching the surface in terms of basic troubleshooting of WinRM-related issues. This will be an evolving document from this starting point, so please check back, also, if you have any questions/suggestions/comments, please let me know below. Thanks!