ArgoCD: transport: Error while dialing: dial tcp: lookup argocd-repo-server: i/o timeout

argocd-repo-server-timeout-tcp.md

ArgoCD Failing to connect to repo-server

I faced this error in my argo cd applicationset controller logs:

argocd-applicationset-controller-d79554d98-qbp8t applicationset-controller time="2024-09-19T10:53:28Z" level=error msg="error generating params" error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup argocd-repo-server: i/o timeout\"" generator="&{0xc000575d40 argocd}"
argocd-applicationset-controller-d79554d98-qbp8t applicationset-controller time="2024-09-19T10:53:28Z" level=error msg="error generating application from params" applicationset=argocd/blogs error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup argocd-repo-server: i/o timeout\"" generator="{nil nil &GitGenerator{RepoURL:https://gitlab.com/x/homelab/gitops.git,Directories:[]GitDirectoryGeneratorItem{GitDirectoryGeneratorItem{Path:apps/blogs/*,Exclude:false,},},Files:[]GitFileGeneratorItem{},Revision:HEAD,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"

And in my repo-server:

argocd-repo-server-797fcbd5c7-lp56l repo-server time="2024-09-19T10:49:23Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2024-09-19T10:49:23Z" grpc.time_ms=0.028 span.kind=server system=grpc

When you go the your repositories in the Argo CD frontend, you will notice theres a timeout trying to fetch the repositories.

Debugging

First we need to see if we can resolve the address of argocd-repo-server from the namespace where argocd is deployed:

kubectl run -it --rm --image=busybox dns-test -n argocd -- nslookup argocd-repo-server
# ;; connection timed out; no servers could be reached

We can see that we cannot resolve addresses, I also tried resolving external addresses which also failed.

The next step is debugging coredns, I have done a rollout restart on that deployment:

kubectl rollout restart deployment/coredns -n kube-system

Once the deployment was restarted, I was able to get a response:

kubectl run -it --rm --image=busybox dns-test -n argocd -- nslookup argocd-repo-server

Response:

Server:		10.43.0.10
Address:	10.43.0.10:53
Name:	argocd-repo-server.argocd.svc.cluster.local
Address: 10.43.82.176

Then the argocd-applicationset-controller was working again.

Thanks

For more argocd content, check out my blog:

• https://ruan.dev/tags/argocd