Running RESTler on the Azure Keyvault service

Create an initial config

Invoke RESTler generate_config, passing a list of the REST API defintion files. This will create a RESTler config file with these files in the SwaggerSpecFilePath.

specs=$(find /Users/mikekistler/Projects/Azure/azure-rest-api-specs/specification/keyvault/resource-manager/Microsoft.KeyVault/stable/2022-07-01 -type f -depth 1)
dotnet $restler_bin/restler/Restler.dll generate_config --specs ${=specs}

Update the dict.json

We need to modify the dict.json to add specific values for certain parameters to be used in the Test phase of Restler. In the case of the Azure App Configuration service, the values needed are:

subscriptionId
resourceGroupName
location
tenantId

Updating dict.json with these values is quite easy with jq:

subscriptionId=$(az account show --query 'id' -o tsv)
location="centralus"
resourceGroupName=$(az group list --query "[?location == 'centralus'] | [?contains(name,'Default')] | [0].name" -o tsv)
tenantId=$(az account show --query 'tenantId' -o tsv)
vars='{
   "subscriptionId": ["'${subscriptionId}'"],
   "resourceGroupName": ["'${resourceGroupName}'"],
   "location": ["'${location}'"],
   "tenantId": ["'${tenantId}'"],
 }'
jq ".restler_custom_payload = $vars" restlerConfig/dict.json > dict.json

Next we need to make a small change to config.json to make CustomDictionaryFilePath to point to our modified dict.json and remove the engine settings and annotations file paths.

This is also easily done with jq.

jq ".CustomDictionaryFilePath = \"$PWD/dict.json\"" restlerConfig/config.json | \
  jq 'del(.EngineSettingsFilePath,.AnnotationFilePath)' > config.json

Create an authentication script

Restler needs a way to obtain authentication tokens to pass on service requests. Use the --token_refresh_command argument to specify a script that will produce authentication tokens. The getToken.sh script does this using the Azure CLI:

#!/bin/bash

find . -name 'token.json' -depth 1 -mtime -1h | grep . &> /dev/null || az account get-access-token > token.json

token=$(jq -r '.accessToken' token.json)

echo "{'user1':{}, 'user2':{}}"
echo "Authorization: bearer ${token}"
echo "Authorization: shadow_unit_test_token"

Compile the configuration

Now we need to run the compile step to pick up these configuration changes.

dotnet $restler_bin/restler/Restler.dll compile config.json

Run RESTler to execute the tests

Now you are ready to run the Test phase of Restler.

dotnet $restler_bin/restler/Restler.dll test --dictionary_file Compile/dict.json --grammar_file Compile/grammar.py --settings Compile/engine_settings.json --token_refresh_command "bash $PWD/getToken.sh" --token_refresh_interval 60

Request coverage (successful / total): 18 / 42
Attempted requests: 29 / 42

Inspect the results

There must be 29-18 = 11 failing tests. How to find them? Start by finding the 'speccov.json' file. Then find the entries in this file that have valid=0 and also have a sample_request.

f=$(find Test -name 'speccov.json')
jq '.[] | select(.valid == 0) | .sample_request // empty ' $f | less

One thing that jumps out quickly is that many of the request_uri's for the failing tests have ?api-version=fuzzstring". These must be operations where the example doesn't specify a value for api-version (or there is no example).

No actually ... these operations do have examples and each example has an api-version value.

HMMM.

mikekistler/restler-keyvault.md