Advanced use of Restler for an Azure Service

This gist will describe a more advanced use of Restler for the Azure App Configuration service. The goal of this effort is to identify ways to improve coverage.

This scenario reuses some of the machinery described in How to create Restler config.

Clone and build Restler

git clone https://github.com/microsoft/restler-fuzzer.git
restler_bin=~/bin/restler
mkdir -p $restler_bin
# Need --python because it can’t find “python” — I use an alias that it did not understand.
python ./build-restler.py --dest_dir $restler_bin --python_path /opt/homebrew/bin/python3

Create the initial Restler config and dict files

restler_bin=~/bin/restler
spec=~/Projects/Azure/azure-rest-api-specs/specification/appconfiguration/resource-manager/Microsoft.AppConfiguration/stable/2022-05-01/appconfiguration.json
dotnet $restler_bin/restler/Restler.dll compile --api_spec $spec

Add values to restler_custom_payload of dict.json

vars='{
   "subscriptionId": ["my-subscription-id"],
   "resourceGroupName": ["my-resource-group"],
   "location": ["my-location"]
}'
jq ".restler_custom_payload = $vars" Compile/dict.json > dict.json

Add annotations to improve dependency calculation

Restler supports annotations to inform its calculation of dependencies.

Based on prior executions of Restler I determined that there are three cases where explicit annotations may help improve the dependency calculation:

The configStoreName passed as a path param to ConfigurationStores_Delete should be used as the configStoreName path param on ConfigurationStores_GetDeleted.
The name of the first element of value in the response body of PrivateLinkResources_ListByConfigurationStore should be passed as the groupName path parameter of PrivateLinkResources_Get.
The id of the first element of value in the response body of ConfigurationStores_ListKeys should be passed as the id field of the request body for ConfigurationStores_RegenerateKey.

Here's the annotations file I created for these (in annotations.json):

{
  "x-restler-global-annotations": [
    {
      "producer_endpoint": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.AppConfiguration/configurationStores/{configStoreName}",
      "producer_method": "DELETE",
      "consumer_endpoint": "/subscriptions/{subscriptionId}/providers/Microsoft.AppConfiguration/locations/{location}/deletedConfigurationStores/{configStoreName}",
      "consumer_method": "GET",
      "producer_resource_name": "configStoreName",
      "consumer_param": "configStoreName"
    },
    {
      "producer_endpoint": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.AppConfiguration/configurationStores/{configStoreName}/privateLinkResources",
      "producer_method": "GET",
      "consumer_endpoint": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.AppConfiguration/configurationStores/{configStoreName}/privateLinkResources/{groupName}",
      "consumer_method": "GET",
      "producer_resource_name": "/value/[0]/name",
      "consumer_param": "groupName"
    },
    {
      "producer_endpoint": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.AppConfiguration/configurationStores/{configStoreName}/listKeys",
      "producer_method": "POST",
      "consumer_endpoint": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.AppConfiguration/configurationStores/{configStoreName}/regenerateKey",
      "consumer_method": "POST",
      "producer_resource_name": "/value/[0]/id",
      "consumer_param": "/id"
    },
  ]
}

Update config.json

We need to update config.json CustomDictionaryFilePath to point to our modified dict.json and AnnotationFilePath to point to our annotations.json:

jq '.CustomDictionaryFilePath = "dict.json" | .AnnotationFilePath = "annotations.json"' Compile/config.json > config.json

Recompile to pick up changes

Run the compile step again to pick up these configuration changes.

dotnet $restler_bin/restler/Restler.dll compile config.json

Run the tests

Now you are ready to run the Test phase of Restler.

dotnet $restler_bin/restler/Restler.dll test --dictionary_file dict.json --grammar_file Compile/grammar.py --settings Compile/engine_settings.json --token_refresh_command "bash $PWD/getToken.sh" --token_refresh_interval 60

I'm now getting:

Request coverage (successful / total): 18 / 24

What is still failing (request_order in parens)?

(1) Operations_List because the request has "api-version=fuzzstring". This operation does not have an example file.
(8) KeyValues_ListByConfigurationStore fails with 404. Known bug -- operation is not implemented.
(11) ConfigurationStores_Get fails with 404. Why?
(16) PrivateEndpointConnections_CreateOrUpdate fails with 400. Known bug -- does not support create.
(17) PrivateEndpointConnections_Get -- not attempted because the Create failed
(18) PrivateEndpointConnections_Delete -- not attempted because the Create failed

Run again and things improve:

Request coverage (successful / total): 19 / 24

Hmmm.

mikekistler/AdvancedRestler.md