micahlee/debugging_info.md

Created September 6, 2019 15:17

Star (1) You must be signed in to star a gist
Fork (1) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/micahlee/d20fcf0a47cfffeaf487f49c7e26b6df.js"></script>
Save micahlee/d20fcf0a47cfffeaf487f49c7e26b6df to your computer and use it in GitHub Desktop.

Download ZIP

Conjur K8s Authenticator Debugging

Raw

debugging_info.md

Display role bindings for conjur-cluster service account token

oc get clusterrolebindings -o json \
  | jq '.items | map(select(any(.subjects[]; .name | contains("conjur-cluster"))))'

Display conjur-authenticator role information

oc describe clusterrole conjur-authenticator

Display configured K8s CA certificate

conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/kubernetes/ca-cert

Verify service account token is what's expected

This does not output the service token itself, but does display the MD5 sum of the token.

TOKEN_SECRET_NAME="$(kubectl get secrets -n <FOLLOWER_NAMESPACE> \
    | grep 'conjur.*service-account-token' \
    | head -n1 \
    | awk '{print $1}')"

# Show MD5 sum for expected token
oc get secret -n <FOLLOWER_NAMESPACE> $TOKEN_SECRET_NAME -o json \
    | jq -r .data.token \
    | base64 --decode \
    | md5sum

# Show MD5 sum for stored token
conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/kubernetes/service-account-token \
    | md5sum

Display configured API URL for authenticator

echo "$(conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/kubernetes/api-url)"

Display generated CA certificate for authenticator

conjur variable value conjur/authn-k8s/<AUTHENTICATOR_ID>/ca/cert

Retrieve metadata about generated CA key:

Conjur show demo:variable:conjur/authn-k8s/<AUTHENTICATOR_ID>/ca/key

List configured authenticators:

curl -ks https://localhost/info | jq .authenticators

whip113 commented Oct 9, 2019 •

edited

Loading

@micahlee
Can we add some common error messages to this as well?

Log from POD:
Starting follower services...
Joined session keyring: 233229515
*** Killing all processes...
Traceback (most recent call last):
File "/sbin/my_init", line 414, in
/tmp/seedfile/start-follower.sh: line 6: 10 Terminated sleep 5
main(args)
File "/sbin/my_init", line 330, in main
import_envvars(False, False)
File "/sbin/my_init", line 90, in import_envvars

If the above is encountered, the service account needs to be allowed to run as root. To correct, run: oc adm policy add-scc-to-user anyuid "system:serviceaccount:$CONJUR_NAMESPACE_NAME:$CONJUR_SERVICEACCOUNT_NAME"

whip113 commented Oct 9, 2019

Another one:

ERROR: 2019/10/03 22:25:34 main.go:53: Failure authenticating: client certificate not found at /etc/conjur/ssl/client.pem

The authenticator doesn't have sufficient privileges through the OCP/K8s API to create the folder/file in the init container for the client certificate. Either create the folder in the container manually, or use the updated seed-fetcher image from Docker Hub.

micah commented Oct 9, 2019 via email

Nathan Whipple <[email protected]> writes:

@micah

wrong micah :)

…

-- micah

Author

micahlee commented Oct 10, 2019

@whip113, these are good additions, thanks! I will add these to the Confluence doc we have for the appliance troubleshooting.

jodyhuntatx commented Oct 24, 2019

Script to verify conjur service account access to K8s API using service-account token, cert & API URL:

#!/bin/bash

CLI=kubectl
CONJUR_NAMESPACE_NAME=cyberark

# Use a cap-D for decoding on Macs
if [[ "$(uname -s)" == "Linux" ]]; then
  BASE64D="base64 -d"
else
  BASE64D="base64 -D"
fi

TOKEN_SECRET_NAME="$($CLI get secrets -n $CONJUR_NAMESPACE_NAME \
    | grep 'conjur.*service-account-token' \
    | head -n1 \
    | awk '{print $1}')"
CERT="$($CLI get secret -n $CONJUR_NAMESPACE_NAME $TOKEN_SECRET_NAME -o json \
      | jq -r '.data["ca.crt"]' \
      | $BASE64D)"
TOKEN="$($CLI get secret -n $CONJUR_NAMESPACE_NAME $TOKEN_SECRET_NAME -o json \
      | jq -r .data.token \
      | $BASE64D)"
API="$($CLI config view --minify -o yaml | grep server | awk '{print $2}')"

echo "$CERT" > k8s.crt
if [[ "$(curl -s --cacert k8s.crt --header "Authorization: Bearer ${TOKEN}" $API/healthz)" == "ok" ]]; then
  echo "Service account access to K8s API verified."
else
  echo
  echo ">>> Service account access to K8s API NOT verified. <<<"
  echo
fi
rm k8s.crt

jralmaraz commented Oct 30, 2019 •

edited

Loading

Thanks for that script @jodyhuntatx ! and for this page all :)

whip113 commented Nov 1, 2019

@micahlee, can we get examples of what good values look like for the variable values we're checking? I've seen in the past where there are double quotes or blank spaces.

jodyhuntatx commented Feb 24, 2022 •

edited

Loading

Client authentication failure. Error in master/leader log was:

CONJ00048I Authentication Error: #<Errors::Authentication::AuthnK8s::MissingClientCertificate: CONJ00029E Client SSL certificate is missing from the header>

The client log showed it was receiving the certificate (login succeeded), but then failing when trying to use the cert for authentication.

DEBUG: 2022/02/24 16:42:55.132874 authenticator.go:267: CAKC039 Trying to log in to Conjur...
DEBUG: 2022/02/24 16:42:55.132881 authenticator.go:158: CAKC041 Logging in as user 'host/test-app'
DEBUG: 2022/02/24 16:42:55.133460 requests.go:22: CAKC045 Login request to: https://t-cdap-follower-svc.test.hcscint.net/authn-k8s/test-cluster/inject_client_cert
DEBUG: 2022/02/24 16:42:55.557182 file.go:54: CAKC051 Waiting for file /etc/conjur/ssl/client.pem to become available...
DEBUG: 2022/02/24 16:42:55.607547 file.go:54: CAKC051 Waiting for file /etc/conjur/ssl/client.pem to become available...
DEBUG: 2022/02/24 16:42:55.607667 authenticator.go:224: CAKC049 Loaded client certificate successfully from /etc/conjur/ssl/client.pem
DEBUG: 2022/02/24 16:42:55.607869 authenticator.go:241: CAKC050 Deleted client certificate from memory
DEBUG: 2022/02/24 16:42:55.607886 authenticator.go:273: CAKC036 Logged in
DEBUG: 2022/02/24 16:42:55.607908 authenticator.go:256: CAKC042 Cert expires: 2022-02-27 16:42:55 +0000 UTC
DEBUG: 2022/02/24 16:42:55.607916 authenticator.go:257: CAKC043 Current date: 2022-02-24 16:42:55.607894465 +0000 UTC
DEBUG: 2022/02/24 16:42:55.607924 authenticator.go:258: CAKC044 Buffer time: 30s
DEBUG: 2022/02/24 16:42:55.608595 requests.go:46: CAKC046 Authn request to: https://t-cdap-follower-svc.test.hcscint.net/authn-k8s/test-cluster/Tst1/host%2Ftest-app/authenticate
DEBUG: 2022/02/24 16:42:55.608621 authenticator.go:318: CAKC069 Sending authn-k8s request...
ERROR: 2022/02/24 16:42:55.658420 main.go:49: CAKC016 Failed to authenticate

Clearly the cert was not reaching the Follower. Turns out the load balancer was terminating TLS and stripping the cert out of the header before sending to the Follower. Configuring the LB for pass-through allowed authentication to succeed.