How does one debug a test-proxy session?

debugging-test-proxy-per-language.md

How to debug test-proxy issues

This writeup is NOT a reference on how to install or run the test-proxy, it is a shortcut guide to helping a test-proxy dev debug each language's tests.

.NET

To run storage tests, azurite must be installed

1. Install npm, ensure node is available on PATH
2. Install npm version of azurite, npm install -g azurite
3. Set environment variable AZURE_AZURITE_LOCATION to the location of your npm root -g output
   • For me this was C:/.tools/.npm-global/

Run the tests

1. Set environment variable PROXY_DEBUG_MODE to true.
   • Optionally, opening up TestProxy.cs, scrolling down the TestProxy startup, and setting debugMode = true will ensure that it always runs with traffic directed at 5000/5001.
2. Start the debugging proxy instance
3. Open VS2022 and then the targeted .NET solution
   • Open VS2022 after running New-TestResources and setting the environment variables in a CLI instance, then open vs manually from there.
   • That is C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/devenv.exe for me.
     • So I just &"C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/devenv.exe"
4. Debug whichever tests by right click -> debug, it'll route to your running proxy.

Seeing proxy output in CI

Set environment variable AZURE_ENABLE_TEST_PROXY_DEBUG_LOGS to true. Something that will bool.TryParse() properly! This way you will be able to check in the actual test output to see what the hell is going wrong on proxy side.

Running tests from command line

Create a props file:

<!-- path/to/propsfile.props -->
<Project>
  <ItemGroup>
    <ProjectReference Include="$(RepoRoot)sdk/storage/Azure.Storage.Common/**/*.csproj" />
  </ItemGroup>
</Project>

Invoke:

dotnet test eng/service.proj `
    --filter "(TestCategory!=Manually) & (TestCategory!=Live) & (Placeholder!=DefaultIgnoreMe)" `
    --framework net8.0 --blame-crash-dump-type full --blame-hang-dump-type full --blame-hang-timeout 60minutes `
    /p:SDKType=all /p:ServiceDirectory=* /p:IncludeSrc=false /p:IncludeSamples=false /p:IncludePerf=false `
    /p:IncludeStress=false /p:RunApiCompat=false /p:InheritDocEnabled=false /p:Configuration=Debug `
    /p:CollectCoverage=true /p:EnableSourceLink=false `
    /p:ProjectListOverrideFile=path/to/propsfile.props `
    /p:EnableOverrideExclusions=true /p:RemoveTrack1Projects=true `
    /p:UseProjectReferenceToAzureClients=false

And for debugging project inclusions:

dotnet msbuild eng/service.proj /t:DumpResolvedProjects  `
    /p:SDKType=all /p:ServiceDirectory=* /p:IncludeSrc=false /p:IncludeSamples=false /p:IncludePerf=false `
    /p:IncludeStress=false /p:RunApiCompat=false /p:InheritDocEnabled=false /p:Configuration=Debug `
    /p:CollectCoverage=true /p:EnableSourceLink=false `
    /p:ProjectListOverrideFile=projectlistoverridefile.props `
    /p:EnableOverrideExclusions=true /p:RemoveTrack1Projects=true `
    /p:UseProjectReferenceToAzureClients=false /v:diag

If you're asking "but what is DumpResolvedProjects?":

<Target Name="DumpResolvedProjects" AfterTargets="ResolveReferences">
  <Message Text="Resolved Projects: @(ProjectReference)" Importance="High" />
</Target>

Add the above target above the GenerateCode target. It's a simple way to see the final project refs.

Python

1. Set environment variable PROXY_MANUAL_START to true

• If you wish to fiddle or otherwise intercept the traffic, using PROXY_URL=http://localhost:5000 is a good idea.

3. Start the debugging proxy instance
   • --storage-location=/path/to/target/repo/root
4. Debug python tests as normal

Go

1. Set the environment variable PROXY_MANUAL_START to true
2. Run the prxoy in selected mode
3. go to targeted directory, go build
4. Invoke tests go test

• go test -run TestClient_GetManifest

For livetests, if there is NO test-resources.json, DO NOT DESPAIR. To successfully invoke livetests in in record mode, I set the following env variables (for sdk/containers/azcontainerregistry), which can be obtained by:

.\eng\common\TestResources\New-TestResources.ps1 containers/azcontainerregistry -UserAuth

${env:REGISTRY_NAME} = '<blahblah>'
${env:AZCONTAINERREGISTRY_RESOURCE_GROUP} = '<my-rg>'
${env:LOGIN_SERVER} = '<blahblah>.azurecr.io'
${env:AZURE_SERVICE_DIRECTORY} = 'AZCONTAINERREGISTRY'
${env:AZCONTAINERREGISTRY_RESOURCE_MANAGER_URL} = 'https://management.azure.com/'
${env:AZCONTAINERREGISTRY_SUBSCRIPTION_ID} = '<subid>'
${env:AZCONTAINERREGISTRY_AZURE_AUTHORITY_HOST} = 'https://login.microsoftonline.com/'
${env:AZCONTAINERREGISTRY_LOCATION} = '<region>'
${env:AZCONTAINERREGISTRY_SERVICE_MANAGEMENT_URL} = 'https://management.core.windows.net/'
#I did NOT set AZCONTAINERREGISTRY_CLOUD, had to reset that to avoid a panic
$env:AZURE_SUBSCRIPTION_ID=$env:AZCONTAINERREGISTRY_SUBSCRIPTION_ID
$env:PROXY_MANUAL_START="true"
$env:AZURE_RECORD_MODE="record"

If you need to make changes to the test framework, and also need to reference that updated local go code. You need to adjust the following.

C:/repo/azure-sdk-for-go [main]|>git diff sdk/containers/azcontainerregistry/go.mod
diff --git a/sdk/containers/azcontainerregistry/go.mod b/sdk/containers/azcontainerregistry/go.mod
index 132a7273c0..8c28f09f81 100644
--- a/sdk/containers/azcontainerregistry/go.mod
+++ b/sdk/containers/azcontainerregistry/go.mod
@@ -25,3 +25,5 @@ require (
        golang.org/x/text v0.14.0 // indirect
        gopkg.in/yaml.v3 v3.0.1 // indirect
 )
+
+replace github.com/Azure/azure-sdk-for-go/sdk/internal => ../../../sdk/internal

The addition of replace is what enables this. Then in recording.go, change the defaultOptions proxyPort to 5001. That way you can get a connection to the proxy port.

C

This language doesn't utilize the proxy. No instructions.

CPP

I've never gotten good help to run shit.

JS

1. PROXY_MANUAL_START=true
2. rush update
3. cd <target package>
4. rush build -t .
5. rushx test:node
6. npx dev-tool run test:vitest --debug-test-proxy -- --typecheck=false test/public/inference/textEmbeddings.spec.ts for specific test?

Java

1. Install maven. Ensure it's available on the path.
2. Set TF_BUILD or CI to true so that the java tests look for an already running proxy.
3. Install necessary prereqs for the package being tested
   • mvn install -f sdk/core/azure-core/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" -DskipTestCompile "-Dcheckstyle.skip=true"
   • mvn install -f sdk/core/azure-core-test/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn install -f common/perf-test-core/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn clean install -f sdk/parents/azure-perf-test-parent/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn install -f sdk/core/azure-core-perf/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn install -f sdk/resourcemanager/ "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
4. mvn surefire:test -f sdk/resourcemanager/azure-resourcemanager-compute/pom.xml -Dtest=TestProxyTests#testRecordWithRedaction
   • -Dtest=<testClassName>#<testFunctionName>

Samples

mvn install -f sdk/containerregistry/azure-containers-containerregistry/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#canUploadDockerManifestWithTagAsync
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#canUploadDockerManifestWithTag
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#getOciListManifest
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#getDockerManifestListType
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#getOciListManifestAsync

If you want to run in record mode, set:

AZURE_TEST_MODE="record"

Get a test-proxy release

url="https://github.com/Azure/azure-sdk-tools/releases/download/Azure.Sdk.Tools.TestProxy_1.0.0-dev.20240515.4/test-proxy-standalone-linux-x64.tar.gz"
directory="/path/to/destination/directory"
mkdir -p "$directory"
wget "$url" -O /tmp/test-proxy-standalone-linux-x64.tar.gz
tar -xzf /tmp/test-proxy-standalone-linux-x64.tar.gz -C "$directory"
rm /tmp/test-proxy-standalone-linux-x64.tar.gz

test-proxy-debugging.md

How to debug test-proxy issues

This writeup is NOT a reference on how to install or run the test-proxy, it is a shortcut guide to helping a test-proxy dev debug each language's tests.

.NET

To run storage tests, azurite must be installed

1. Install npm, ensure node is available on PATH
2. Install npm version of azurite, npm install -g azurite
3. Set environment variable AZURE_AZURITE_LOCATION to the location of your npm root -g output
   • For me this was C:/.tools/.npm-global/

Run the tests

1. Set environment variable PROXY_DEBUG_MODE to true.
   • Optionally, opening up TestProxy.cs, scrolling down the TestProxy startup, and setting debugMode = true will ensure that it always runs with traffic directed at 5000/5001.
2. Start the debugging proxy instance
3. Open VS2022 and then the targeted .NET solution
   • Open VS2022 after running New-TestResources and setting the environment variables in a CLI instance, then open vs manually from there.
   • That is C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/devenv.exe for me.
     • So I just &"C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/devenv.exe"
4. Debug whichever tests by right click -> debug, it'll route to your running proxy.

Seeing proxy output in CI

Set environment variable AZURE_ENABLE_TEST_PROXY_DEBUG_LOGS to true. Something that will bool.TryParse() properly! This way you will be able to check in the actual test output to see what the hell is going wrong on proxy side.

Running tests from command line

Create a props file:

<!-- path/to/propsfile.props -->
<Project>
  <ItemGroup>
    <ProjectReference Include="$(RepoRoot)sdk/storage/Azure.Storage.Common/**/*.csproj" />
  </ItemGroup>
</Project>

Invoke:

dotnet test eng/service.proj `
    --filter "(TestCategory!=Manually) & (TestCategory!=Live) & (Placeholder!=DefaultIgnoreMe)" `
    --framework net8.0 --blame-crash-dump-type full --blame-hang-dump-type full --blame-hang-timeout 60minutes `
    /p:SDKType=all /p:ServiceDirectory=* /p:IncludeSrc=false /p:IncludeSamples=false /p:IncludePerf=false `
    /p:IncludeStress=false /p:RunApiCompat=false /p:InheritDocEnabled=false /p:Configuration=Debug `
    /p:CollectCoverage=true /p:EnableSourceLink=false `
    /p:ProjectListOverrideFile=path/to/propsfile.props `
    /p:EnableOverrideExclusions=true /p:RemoveTrack1Projects=true `
    /p:UseProjectReferenceToAzureClients=false

And for debugging project inclusions:

dotnet msbuild eng/service.proj /t:DumpResolvedProjects  `
    /p:SDKType=all /p:ServiceDirectory=* /p:IncludeSrc=false /p:IncludeSamples=false /p:IncludePerf=false `
    /p:IncludeStress=false /p:RunApiCompat=false /p:InheritDocEnabled=false /p:Configuration=Debug `
    /p:CollectCoverage=true /p:EnableSourceLink=false `
    /p:ProjectListOverrideFile=projectlistoverridefile.props `
    /p:EnableOverrideExclusions=true /p:RemoveTrack1Projects=true `
    /p:UseProjectReferenceToAzureClients=false /v:diag

If you're asking "but what is DumpResolvedProjects?":

<Target Name="DumpResolvedProjects" AfterTargets="ResolveReferences">
  <Message Text="Resolved Projects: @(ProjectReference)" Importance="High" />
</Target>

Add the above target above the GenerateCode target. It's a simple way to see the final project refs.

Python

1. Set environment variable PROXY_MANUAL_START to true

• If you wish to fiddle or otherwise intercept the traffic, using PROXY_URL=http://localhost:5000 is a good idea.

3. Start the debugging proxy instance
   • --storage-location=/path/to/target/repo/root
4. Debug python tests as normal

Go

1. Set the environment variable PROXY_MANUAL_START to true
2. Run the prxoy in selected mode
3. go to targeted directory, go build
4. Invoke tests go test

• go test -run TestClient_GetManifest

For livetests, if there is NO test-resources.json, DO NOT DESPAIR. To successfully invoke livetests in in record mode, I set the following env variables (for sdk/containers/azcontainerregistry), which can be obtained by:

.\eng\common\TestResources\New-TestResources.ps1 containers/azcontainerregistry -UserAuth

${env:REGISTRY_NAME} = '<blahblah>'
${env:AZCONTAINERREGISTRY_RESOURCE_GROUP} = '<my-rg>'
${env:LOGIN_SERVER} = '<blahblah>.azurecr.io'
${env:AZURE_SERVICE_DIRECTORY} = 'AZCONTAINERREGISTRY'
${env:AZCONTAINERREGISTRY_RESOURCE_MANAGER_URL} = 'https://management.azure.com/'
${env:AZCONTAINERREGISTRY_SUBSCRIPTION_ID} = '<subid>'
${env:AZCONTAINERREGISTRY_AZURE_AUTHORITY_HOST} = 'https://login.microsoftonline.com/'
${env:AZCONTAINERREGISTRY_LOCATION} = '<region>'
${env:AZCONTAINERREGISTRY_SERVICE_MANAGEMENT_URL} = 'https://management.core.windows.net/'
#I did NOT set AZCONTAINERREGISTRY_CLOUD, had to reset that to avoid a panic
$env:AZURE_SUBSCRIPTION_ID=$env:AZCONTAINERREGISTRY_SUBSCRIPTION_ID
$env:PROXY_MANUAL_START="true"
$env:AZURE_RECORD_MODE="record"

If you need to make changes to the test framework, and also need to reference that updated local go code. You need to adjust the following.

C:/repo/azure-sdk-for-go [main]|>git diff sdk/containers/azcontainerregistry/go.mod
diff --git a/sdk/containers/azcontainerregistry/go.mod b/sdk/containers/azcontainerregistry/go.mod
index 132a7273c0..8c28f09f81 100644
--- a/sdk/containers/azcontainerregistry/go.mod
+++ b/sdk/containers/azcontainerregistry/go.mod
@@ -25,3 +25,5 @@ require (
        golang.org/x/text v0.14.0 // indirect
        gopkg.in/yaml.v3 v3.0.1 // indirect
 )
+
+replace github.com/Azure/azure-sdk-for-go/sdk/internal => ../../../sdk/internal

The addition of replace is what enables this. Then in recording.go, change the defaultOptions proxyPort to 5001. That way you can get a connection to the proxy port.

C

This language doesn't utilize the proxy. No instructions.

CPP

I've never gotten good help to run shit.

JS

TODO: reference test-proxy livetests as this information using rush is out of date.

1. PROXY_MANUAL_START=true
2. rush update
3. cd <target package>
4. rush build -t .
5. rushx test:node
6. npx dev-tool run test:vitest --debug-test-proxy -- --typecheck=false test/public/inference/textEmbeddings.spec.ts for specific test?

Java

1. Install maven. Ensure it's available on the path.
2. Set TF_BUILD or CI to true so that the java tests look for an already running proxy.
3. Install necessary prereqs for the package being tested
   • mvn install -f sdk/core/azure-core/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" -DskipTestCompile "-Dcheckstyle.skip=true"
   • mvn install -f sdk/core/azure-core-test/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn install -f common/perf-test-core/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn clean install -f sdk/parents/azure-perf-test-parent/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn install -f sdk/core/azure-core-perf/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
   • mvn install -f sdk/resourcemanager/ "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
4. mvn surefire:test -f sdk/resourcemanager/azure-resourcemanager-compute/pom.xml -Dtest=TestProxyTests#testRecordWithRedaction
   • -Dtest=<testClassName>#<testFunctionName>

Samples

mvn install -f sdk/containerregistry/azure-containers-containerregistry/pom.xml "-Dcodesnippet.skip" "-Drevapi.skip" "-Dspotbugs.skip" -DskipTests=true "-Djacoco.skip" "-Dmaven.javadoc.skip=true" "-Dcheckstyle.skip=true"
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#canUploadDockerManifestWithTagAsync
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#canUploadDockerManifestWithTag
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#getOciListManifest
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#getDockerManifestListType
mvn surefire:test -f sdk/containerregistry/azure-containers-containerregistry/pom.xml -Dtest=ContainerRegistryContentClientIntegrationTests#getOciListManifestAsync

If you want to run in record mode, set:

AZURE_TEST_MODE="record"

Get a test-proxy release

url="https://github.com/Azure/azure-sdk-tools/releases/download/Azure.Sdk.Tools.TestProxy_1.0.0-dev.20240515.4/test-proxy-standalone-linux-x64.tar.gz"
directory="/path/to/destination/directory"
mkdir -p "$directory"
wget "$url" -O /tmp/test-proxy-standalone-linux-x64.tar.gz
tar -xzf /tmp/test-proxy-standalone-linux-x64.tar.gz -C "$directory"
rm /tmp/test-proxy-standalone-linux-x64.tar.gz

Running test-proxy on windows with tests from WSL

To do this, you need to make certain the following bases are covered. The following applies to WSL2 only. I don't use WSL1 anymore.

• On windows side, ensure that the proxy is bound to all interfaces.
  • start <other arguments> -- --urls="http://0.0.0.0:5000"
• Then on linux side, ensure you're talking through the windows loopback IP.

  • semick@CPC-scbed-QJJWS:~/repo/azure-sdk-for-python/sdk/communication/azure-communication-sms$ fix-communication-314-tests|>WIN_IP=$(ip route | awk '/^default/ {print $3; exit}')
    echo "$WIN_IP"
    172.17.112.1
    curl "http://$WIN_IP:5000/Info/Available
    ...

  • The key is that you use PROXY_URL equal to whatever IP is indicated by WIN_IP, NOT localhost.

export CERT_LOCATION=/home/semick/repo/azure-sdk-for-python/.certificate
export SSL_CERT_FILE=$CERT_LOCATION/dotnet-devcert.pem
export REQUESTS_CA_BUNDLE=$SSL_CERT_FILE