Skip to content

Instantly share code, notes, and snippets.

@juandspy

juandspy/README.md

Last active June 26, 2025 07:35
Show Gist options
  • Save juandspy/22661ef4e81f261bee5740c428edfe2e to your computer and use it in GitHub Desktop.
Save juandspy/22661ef4e81f261bee5740c428edfe2e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
README.md

Disconnected Insights on ACM

The goal of this PoC is to enable Insights Operator on a disconnected cluster.

flowchart LR
    subgraph Managed Cluster
    IO[Insights Operator]
    end
    subgraph Hub Cluster
    IP[Insights Proxy]
    end
    IO --> IP
    IP --> CRC[console.redhat.com]
Loading

This is useful when the managed cluster is disconnected from the internet but has access to the Hub Cluster.

Prerequisites

  • A Hub Cluster with the ACM operator installed
  • A managed cluster (be it local, on AWS, etc.)

Insights Proxy

The first step is to install the Insights Proxy on the Hub Cluster:

oc import-image insights-proxy/insights-proxy-container-rhel9:1.5.3 --from=registry.redhat.io/insights-proxy/insights-proxy-container-rhel9:1.5.3 --confirm

oc apply -f insights-proxy.yaml

You can now check the Insights Proxy on oc get service/insights-proxy-service -o json | jq -r '.status.loadBalancer.ingress[0].hostname':

curl -s --proxy "http://$(oc get service/insights-proxy-service -o json | jq -r '.status.loadBalancer.ingress[0].hostname'):80" https://console.redhat.com/api/insights-results-aggregator/v2/info | jq

Insights Operator

Now you can jump to your managed cluster and configure the IO to use the proxy. You need to patch the clusterversion/version object:

oc patch clusterversion/version --type merge -p '{"spec":{"capabilities":{"additionalEnabledCapabilities":["openshift-samples","marketplace","Console","MachineAPI","ImageRegistry","DeploymentConfig","Build","OperatorLifecycleManager","Ingress", "Insights"]}}}'

And then configure the Insights Operator to use the Insights Proxy (oc apply -f io-config.yaml). Please update this file httpsProxy key with the result of

echo "http://$(oc get service/insights-proxy-service -o json | jq -r '.status.loadBalancer.ingress[0].hostname'):80"

Then, restart the operator in order to force a new data gathering with the latest configuration:

oc delete pod/$(oc get pods | grep insights-operator | cut -d' ' -f1)

Optional: disconnected pod

In order to check that this solution would work on a disconnected cluster, you can create a network policy to block all the internet access except for the Insights Proxy URL (IPs).

  1. List the IPs of the Insights Proxy:
dig +short ad2f2703e38c64ddf933371a8885293b-1014996582.us-east-1.elb.amazonaws.com
  1. Create a network policy. Replace the IPs with yours: oc apply -f io-network-policy.yaml.
  2. Check that the Insights Operator is not able to connect to the internet:
> oc rsh $(oc get pods | grep insights-operator | cut -d' ' -f1)
$ curl -s https://console.redhat.com/api/insights-results-aggregator/v2/info | jq

this should timeout. 4. Then, restart the operator:

oc delete pod/$(oc get pods | grep insights-operator | cut -d' ' -f1)
  1. And check the logs:
I0626 07:12:29.838899 1 observer_polling.go:159] Starting file observer
W0626 07:12:54.843969 1 builder.go:266] unable to get owner reference (falling back to namespace): Get "https://10.217.4.1:443/api/v1/namespaces/openshift-insights/pods/insights-operator-5db75bbd5f-cbrz5": dial tcp 10.217.4.1:443: i/o timeout
E0626 07:13:20.106306 1 start.go:245] error initializing delegating authentication: unable to load configmap based request-header-client-ca-file: Get "https://10.217.4.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.217.4.1:443: i/o timeout

This means that the own cluster IPs are blocked, so you may need to add them to the network policy and restart the operator again.

You may need to add more IPs to the network policy:

❯   oc get endpoints kubernetes -n default
NAME         ENDPOINTS             AGE
kubernetes   192.168.126.11:6443   101d
❯ oc get svc kubernetes -n default
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.217.4.1   <none>        443/TCP   101d

I've replace the latter with 10.217.0.0/16 and the former with 192.168.126.11/32. 6. Apply the new network policy and restart the operator again. 7. Check the logs:

❯ oc logs --all-containers -f pod/$(oc get pods | grep insights-operator | cut -d' ' -f1) | grep "Successfully reported"
Raw
insights-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: insights-proxy
labels:
app: insights-proxy
spec:
replicas: 1
selector:
matchLabels:
app: insights-proxy
template:
metadata:
labels:
app: insights-proxy
spec:
initContainers:
- name: copy-nginx-data
# Due to the way the image is built and Openshift permissions,
# the /opt/app-root/nginx/etc/nginx is not writable so we need to mount
# an empty path. However, that deletes all the files there, so we need
# to first copy them to the empty volume.
# This is because the image uses an nginx user (1001) to own the files
image: registry.redhat.io/insights-proxy/insights-proxy-container-rhel9:1.5.3
command:
- /bin/sh
- -c
- >
cp -r /opt/app-root/nginx/* /nginx-data/
volumeMounts:
- name: nginx-data
mountPath: /nginx-data
containers:
- name: insights-proxy
image: registry.redhat.io/insights-proxy/insights-proxy-container-rhel9:1.5.3
volumeMounts:
- mountPath: /opt/app-root/certs
name: certs
- mountPath: /opt/app-root/nginx
name: nginx-data
- name: rhproxy-env
mountPath: /opt/app-root/rhproxy-env/
env:
- name: RHPROXY_SERVICE_PORT
value: "3128"
volumes:
- name: certs
emptyDir: {}
- name: nginx-data
emptyDir: {}
- name: rhproxy-env
configMap:
name: rhproxy-env
items:
- key: redhat.servers
path: redhat.servers
---
apiVersion: v1
kind: Service
metadata:
name: insights-proxy-service
labels:
app: insights-proxy
spec:
type: LoadBalancer
selector:
app: insights-proxy
ports:
- name: rhproxy-service
protocol: TCP
port: 80
targetPort: 3128
---
apiVersion: v1
kind: ConfigMap
metadata:
name: rhproxy-env
data:
redhat.servers: |
# my custom servers
api.openshift.com
# RedHat Insights Servers
api.access.redhat.com
cert-api.access.redhat.com
console.redhat.com
cert.console.redhat.com
cloud.redhat.com
cert.cloud.redhat.com
connect.cloud.redhat.com
subscription.rhsm.redhat.com
sso.redhat.com
# Dnf/Yum RedHat Servers
cdn.redhat.com
mirrors.fedoraproject.org
mirrormanager.fedoraproject.org
codecs.fedoraproject.org
Raw
io-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: insights-config
namespace: openshift-insights
data:
config.yaml: |
dataReporting:
interval: 0h1m0s
uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload
storagePath: /var/lib/insights-operator
downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports
conditionalGathererEndpoint: https://console.redhat.com/api/gathering/gathering_rules
obfuscation: [workload_names networking]
sca:
disabled: false
endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates
interval: 0h1m0s
alerting:
disabled: false
clusterTransfer:
endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/
interval: 0h1m0s
proxy:
httpsProxy: CHANGEME
Raw
io-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: restrict-egress
namespace: openshift-insights
spec:
podSelector: {}
policyTypes:
- Egress
egress:
# Allow all in-cluster service traffic
- to:
- ipBlock:
cidr: 10.217.0.0/16
# Allow to the API server endpoint (node IP): oc get endpoints kubernetes -n default
- to:
- ipBlock:
cidr: 192.168.126.11/32
ports:
- protocol: TCP
port: 6443
# Allow to the ELB IPs
- to:
- ipBlock: { cidr: 3.212.39.88/32 }
- ipBlock: { cidr: 54.235.186.191/32 }
- ipBlock: { cidr: 34.238.109.97/32 }
- ipBlock: { cidr: 34.192.149.215/32 }
- ipBlock: { cidr: 44.195.213.173/32 }
ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 443
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment