Skip to content

Instantly share code, notes, and snippets.

@thoughtful-solutions

thoughtful-solutions/Azure-Cognitive-Search-on-JSON-in-Blob-Storage.md

Last active September 2, 2020 15:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save thoughtful-solutions/ec7f6ef614e43a23946d6758f1beb6b1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save thoughtful-solutions/ec7f6ef614e43a23946d6758f1beb6b1 to your computer and use it in GitHub Desktop.
Download ZIP
Azure-Cognitive-Services-on-JSON-in-Blob-Storage
Raw
Azure-Cognitive-Search-on-JSON-in-Blob-Storage.md

Azure Congnative Search

Azure Cognative Search is a search tool capable of ingesting and searching a number of data formats including Office and JSON documents.

The service is currently under development and marked as preview This means that the azure CLI does not provide all the tooling you might wish and you are required to post some configuration pieces using curl or similar.

Build Search Service

The free tier of service (SKU) is limited in the size of the documents it can handle.

   az search service create --name mtidev000ss --resource-group mtidev000rg --sku FREE --location westeurope

Get API Key from Search Service

   az search admin-key show --resource-group mtidev000rg --service-name mtidev000ss

Create an Index for the Search Service for a JSON document

An index needs creating, it requires the name of the search service mitdev000ss and the API Key returned from the previous command line.

You can choose the name of your own index, in this example we choose mti-results2 ad this value is used both in the API URL and in the JSON data associated with the call. They must be identicle for the call to work.

When creating your index you must specify exactly what fields exist within your JSON data structure,

  • "name" : the name of the JSON tag
  • "type" : the data type of the associate value (Edm.String,Edm.Double,Edm.Int32,Edm.ComplexType)
  • "searchable" : whether the values of the tag are to be searchable
  • "sortable" : whether the values of the tag are to be sortable
  • "filterable" : whether the values of the tag are to be filterable
  • "key" : whether the value is a primary key
curl --location --request PUT 'https://mtidev000ss.search.windows.net/indexes/mti-results2?api-version=2020-06-30' \
     --header 'Content-Type: application/json' \
     --header 'api-key: 33D1E0AE361F57FC55CEDFA573B6C794' \
     --data '{
    "name": "mti-results2", 
    "fields": [
        {"name": "enterpriseId", "type": "Edm.String", "searchable": true, "filterable": true},
        {"name": "apiAuthKey", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": true, "facetable": false},
        {"name": "applyNo", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": true, "facetable": false, "key": true},
        {"name": "ekycUserId", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": false, "facetable": false},
        {"name": "ipAddress", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": false, "facetable": false},
        {"name": "userAgent", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": false, "facetable": false},
        {"name": "ekycStartDate", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": true},
        {"name": "ekycEndDate", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": false},
        {"name": "docType", "type": "Edm.Int32", "filterable": true, "sortable": true, "facetable": true},
        {"name": "identificationType", "type": "Edm.Int32", "filterable": true, "sortable": true, "facetable": true},
        {"name": "matchingResult", "type": "Edm.ComplexType",
        "fields": [
        {"name": "matchingCount", "type": "Edm.Int32", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
        {"name": "docFrontMatch", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": true, "facetable": true},
        {"name": "docTiltedMatch", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": true, "facetable": true},
        {"name": "docLiveMatch", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": true, "facetable": true},
        {"name": "docFrontScore", "type": "Edm.Double", "searchable": false, "filterable": true, "sortable": true, "facetable": true},
        {"name": "docTiltedScore", "type": "Edm.Double", "searchable": false, "filterable": true, "sortable": true, "facetable": true},
        {"name": "docLiveScore", "type": "Edm.Double", "searchable": false, "filterable": true, "sortable": true, "facetable": true}
        ]
        },
        {"name": "imageResult", "type": "Edm.ComplexType",
          "fields": [
          {"name": "docFrontImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
          {"name": "docFrontTrimImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
          {"name": "docBackImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
          {"name": "docBackTrimImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
          {"name": "docTiltedImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
          {"name": "docTiltedTrimImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false},
          {"name": "selfieImage", "type": "Edm.String", "filterable": false, "sortable": false, "facetable": false, "searchable": false}
          ]
     }
  ]
}'

Create data source in Search Service

It is important you get the API key

  az search admin-key show --resource-group mtidev000rg --service-name mtidev000ss

and the connection string

  az storage account show-connection-string  --resource-group mtidev000rg --name mtidev000sa

The API key is placed in the header and the connection-string is placed in the JSON data

you are creating a new datastore and you need to specify the container

curl --location --request POST 'https://mtidev000ss.search.windows.net/datasources?api-version=2020-06-30' \
--header 'Content-Type: application/json' \
--header 'api-key: 33DXXXXXXXXXXB6C794' \
--data-raw '    {
        "name" : "mtidev000ds",
        "type" : "azureblob",
        "credentials" : { "connectionString" : "DefaultEndpointsProtocol=https;AccountName=mtidev000sa;AccountKey=tg05an1Mv7o68QVto/ERvUvIzzsKVHlXnrM551MXdvHwPsrg4UHVJxOYcYnETlC+hC5/7mbSxcCLSUic52S+uA==;EndpointSuffix=core.windows.net" },
        "container" : { "name" : "mtidev000sc" }
    }  '

Create indexer (and executes on first instantiation)

curl --location --request POST 'https://mtidev000ss.search.windows.net/indexers?api-version=2020-06-30' \
--header 'Content-Type: application/json' \
--header 'api-key: 262E16699D039339A4C3C1FEEACA3CD4' \
--data-raw '{  
    "name" : "mtidev000ixr", 
    "dataSourceName" : "mtidev000ds", 
    "targetIndexName" : "mti-results2",
    "parameters" : { "configuration" : { "parsingMode" : "json" } }
}  '

Get the Indexer to run again

curl --location --request POST 'https://mtidev000ss.search.windows.net/indexers/mtidev000ixr/run?api-version=2020-06-30' \
--header 'Content-Type: application/json' \
--header 'api-key: 262xxxxxxx4' \
--data-raw '{   
    "name" :  "mtidev000ixr",  
    "dataSourceName" : "mtidev000ds",  
    "targetIndexName" : "mti-results2"
}  '

Search Documents

You can return all the documents in the search set with

curl --location --request GET 'https://mtidev000ss.search.windows.net/indexes/mti-results2/docs?search=*&$count=true&api-version=2020-06-30' \
--header 'Content-Type: application/json' \
--header 'api-key: 262E1669xxxxxxCA3CD4'

or you can be more selective

curl --location --request GET 'https://mtidev000ss.search.windows.net/indexes/mti-results2/docs?search=*&$count=true&$filter=applyNo%20eq%20%2700000007%27&$select=applyNo,ekycStartDate&api-version=2020-06-30' \
--header 'Content-Type: application/json' \
--header 'api-key: 262E1669xxxxxxx3CD4'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment