golang, delve, rr, openshift-installer

compile.patch

diff --git a/hack/build.sh b/hack/build.sh
index c5b9156ed3..0c157442bf 100755
--- a/hack/build.sh
+++ b/hack/build.sh
@@ -52,7 +52,7 @@ export CGO_ENABLED=0
 
 case "${MODE}" in
 release)
-	LDFLAGS="${LDFLAGS} -s -w"
+	#LDFLAGS="${LDFLAGS}"
 	TAGS="${TAGS} release"
 	if test "${SKIP_GENERATION}" != y
 	then
@@ -73,4 +73,4 @@ then
 fi
 
 # shellcheck disable=SC2086
-go build ${GOFLAGS} -ldflags "${LDFLAGS}" -tags "${TAGS}" -o "${OUTPUT}" ./cmd/openshift-install
+go build ${GOFLAGS} -gcflags "all=-N -l" -ldflags "${LDFLAGS}" -tags "${TAGS}" -o "${OUTPUT}" ./cmd/openshift-install
diff --git a/terraform/Makefile b/terraform/Makefile
index 524b550535..9d3def5891 100644
--- a/terraform/Makefile
+++ b/terraform/Makefile
@@ -7,7 +7,8 @@ GO_BUILD_TARGETS:=	$(foreach DIR,$(TFSUBDIRS), $(subst $(DIR),go-build.$(DIR),$(
 GO_CLEAN_TARGETS:=	$(foreach DIR,$(TFSUBDIRS), $(subst $(DIR),go-clean.$(DIR),$(DIR)))
 TERRAFORM_PROVIDER_TARGETS := $(foreach DIR,$(TFSUBDIRS), bin/$(TARGET_OS_ARCH)/terraform-provider-$(DIR).zip)
 
-LDFLAGS:= "-s -w"
+LDFLAGS:= ""
+GCFLAGS:= "all=-N -l"
 
 .PHONY: all
 all: go-build
@@ -26,7 +27,7 @@ $(GO_BUILD_TARGETS): go-build.%: bin/$(TARGET_OS_ARCH)/terraform-provider-%.zip
 $(TERRAFORM_PROVIDER_TARGETS): bin/$(TARGET_OS_ARCH)/terraform-provider-%.zip: providers/%/go.mod
 	cd providers/$*; \
 	if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $$2 }'|sed 's|"||g'`; fi; \
-	go build -ldflags $(LDFLAGS) -o ../../bin/$(TARGET_OS_ARCH)/terraform-provider-$* "$$path"; \
+	go build -gcflags $(GCFLAGS) -ldflags $(LDFLAGS) -o ../../bin/$(TARGET_OS_ARCH)/terraform-provider-$* "$$path"; \
 	zip -1j ../../bin/$(TARGET_OS_ARCH)/terraform-provider-$*.zip ../../bin/$(TARGET_OS_ARCH)/terraform-provider-$*;
 
 .PHONY: go-build-terraform
@@ -34,7 +35,7 @@ go-build-terraform: bin/$(TARGET_OS_ARCH)/terraform
 
 bin/$(TARGET_OS_ARCH)/terraform: terraform/go.mod
 	cd terraform; \
-	go build -ldflags $(LDFLAGS) -o ../bin/$(TARGET_OS_ARCH)/terraform ./vendor/github.com/hashicorp/terraform
+	go build -gcflags $(GCFLAGS) -ldflags $(LDFLAGS) -o ../bin/$(TARGET_OS_ARCH)/terraform ./vendor/github.com/hashicorp/terraform
 
 .PHONY: go-clean
 go-clean: go-clean-providers go-clean-terraform
@@ -45,7 +46,7 @@ $(GO_CLEAN_TARGETS): go-clean.%:
 
 go-clean-providers:
 	rm -f bin/*/terraform-provider-*
-	
+
 go-clean-terraform:
 	rm -f bin/*/terraform
 

howto.md

Using rr+delve to debug openshift installer

Steps

1 - Developer: Compile Installer

Apply patch provided in this gist and build installer

# Assuming fedora, centos, rhel
# install golang and rr
sudo dnf install golang rr -y
# Install delve
go install github.com/go-delve/delve/cmd/dlv@latest
# Clone installer
git clone https://github.com/openshift/installer
# Apply changes to hack/build.sh and terraform/Makefile
git apply ../compile.patch
# Compile installer
./hack/build.sh

2 - Developer and Enduser: Provide installer binary, install rr

sudo dnf install rr -y
# OR
wget https://github.com/rr-debugger/rr/releases/download/5.6.0/rr-5.6.0-Linux-$(uname -m).rpm
sudo dnf install rr-5.6.0-Linux-$(uname -m).rpm

3 - Enduser: Run installer with rr record

# required kernel settings
sudo sysctl kernel.perf_event_paranoid=-1
sudo sysctl kernel.kptr_restrict=0
# run openshift-install with rr record
rr record openshift-install create cluster --log-level debug
# rr pack gets trace ready for transport
rr pack
tar -cvzf trace.tar.gz ${HOME}/.local/share/rr/

4 - Developer: Debug

# extract trace
tar -xvf trace.tar.gz
# Required kernel parameters
sudo sysctl kernel.perf_event_paranoid=-1
sudo sysctl kernel.kptr_restrict=0
# Run delve with rr backend
dlv replay --backend=rr --headless --listen=:2345 --api-version=2 --accept-multiclient ${HOME}/.local/share/rr/openshift-install-0
# debug from workstation

Enduser and Developer Caveats

• CPU architectures must be similar
• If you are using vSphere virtualization you must enable Performance counters. In the guest configuration navigate to vCPU -> Performance Counters, check Enable virtualized CPU performance counters
• rr saves the execution of the installer. All data structures including authentication and api objects are available to view. The transported trace should be encrypted.

Links

• https://rr-project.org/
• https://github.com/rr-debugger/rr
• https://www.youtube.com/watch?v=sMnw28M-fMg
• https://devconfcz2020a.sched.com/event/YOsC/deterministic-debugging-with-delve

Old How-To

On all machines involved: debug or running openshift-install

Install rr, golang and delve

The machines that will run and/or debug the installer must have rr installed. If you are running fedora that can be done as easily as:

sudo dnf install golang rr -y
go install github.com/go-delve/delve/cmd/dlv@latest

Otherwise:

wget https://github.com/rr-debugger/rr/releases/download/5.6.0/rr-5.6.0-Linux-$(uname -m).rpm
sudo dnf install rr-5.6.0-Linux-$(uname -m).rpm

Kernel settings

sudo sysctl kernel.perf_event_paranoid=-1
sudo sysctl kernel.kptr_restrict=0

VMware vSphere Virtual Machine settings

If you are using vSphere virtualization you must enable Performance counters.

In the guest configuration navigate to vCPU -> Performance Counters, check Enable virtualized CPU performance counters

Changes to installer build

git clone https://github.com/openshift/installer

Open ./hack/build.sh

Comment out

LDFLAGS="${LDFLAGS} -s -w"

Add -gcflags "all=-N -l" to go build

go build ${GOFLAGS} -gcflags "all=-N -l" -ldflags "${LDFLAGS}" -tags "${TAGS}" -o "${OUTPUT}" ./cmd/openshift-install

Rebuild the installer

./hack/build.sh

This doesn't change the terraform provider builds, that would need to be investigated.

Execute the installer with rr

rr record ../openshift-install create cluster --log-level debug

If the trace is to be used on an alternate machine:

rr pack
tar -cvzf trace.tar.gz ${HOME}/.local/share/rr/

The ./rr directory will take significant amount of disk space if rr pack is not ran.

Debugging with goland and delve replay

In my limited experience it seems CPU models will matter so most likely you will be remotely executing delve.

Disable the firewall (yeah I am lazy)

sudo systemctl stop firewalld
sudo systemctl mask firewalld

Start delve in replay mode. Change the directory if the trace is not in the default folder.

dlv replay --backend=rr --headless --listen=:2345 --api-version=2 --accept-multiclient ${HOME}/.local/share/rr/openshift-install-0

Keep this window near by, the installer will still emit the typical output.

In goland, Add Configuration -> Add New -> Go Remote. Change Host to where delve is running.

Debug as normal...