OSDU R2 Setup Instructions

Create ADO and Import Projects for R2

This typically takes about 10-15 minutes to complete.

Create a new ADO Project in your organization called osdu-r2
In the base project repo now import the base project
- https://dev.azure.com/osdu-demo/osdu-r2/_git/osdu-r2
Create Empty Repositories (No Readme)
- osdu-infrastructure
- entitlements-azure
- legal
- indexer-queue
- storage
- indexer
- search

Setup the Variable Group Neccessary for the Pipeline

Variable	Value
ACCESS_TOKEN	<your_personal_access_token>
INFRASTRUCTURE_REPO	https://dev.azure.com/osdu-demo/osdu-r2/_git/osdu-infrastructure
ENTITLEMENTS_REPO	https://dev.azure.com/osdu-demo/osdu-r2/_git/entitlements-azure
LEGAL_REPO	https://dev.azure.com/osdu-demo/osdu-r2/_git/legal

Add a Pipeline osdu-r2-repository-sync --> azure-pipelines.yml
Execute the Pipeline which will then pull the required code into the ADO project repos.

Build osdu-infrastructure for R2

This typically takes about 2 hours to complete.

Prerequisites

Here is an Azure Virtual Developer Machine that can be used if necessary.

Procedures are tested using Ubuntu within WSL for Windows 10. (Typically MacOS works well)

Clone Infrastructure

Clone the osdu-infrastructure repository to a local machine.

Execute Install Script

The script ./scripts/install.sh will conveniently setup the common things that are necessary to execute a pipeline.

Login to the azure cli and set the default account to the desired subscription.
Follow the instructions for bootstraping the osdu-infrastructure pipeline located in the README.md of that project space.

Installed Common Resources

Resource Group
Storage Account
Key Vault
Applications for Integration Testing (2)

Elastic Search Setup

Infrastructure assumes bring your own Elastic Search Instance at a version of 6.8.3 and access information must be stored in the Common KeyVault.

AZURE_VAULT="<your_keyvault>"
az keyvault secret set --vault-name $AZURE_VAULT --name "elastic-endpoint-ado-demo" --value <your_es_endpoint>
az keyvault secret set --vault-name $AZURE_VAULT --name "elastic-username-ado-demo" --value <your_es_username>
az keyvault secret set --vault-name $AZURE_VAULT --name "elastic-password-ado-demo" --value <your_es_password>

# This command will extract all Key Vault Secrets
for i in `az keyvault secret list --vault-name $AZURE_VAULT --query [].id -otsv`
do
   echo "export ${i##*/}=\"$(az keyvault secret show --vault-name $AZURE_VAULT --id $i --query value -otsv)\""
done

Configure Azure DevOps Service Connection

Configure an ARM Resources Service Connection for the desired subscription.
- Scope should be to the desired Subscription but do not apply scope to a Resource Group
Locate the Service Principal created () in Azure Active Directory and elevate the principal capability by adding in 2 API Permissions
- Azure Active Directory Graph - Application.ReadWrite.OwnedBy
- Microsoft Graph - Application.ReadWrite.OwnedBy

These 2 API's require Grant Admin Consent

In Azure Portal locat the subscription and under Access control (IAM) add an Owner Role Assignment to the principal then remove the default created Contributor role.

Setup ADO required Libraries

Setup and Configure the ADO Library Infrastructure Pipeline Variables

Variable	Value
AGENT_POOL	Hosted Ubuntu 1604
BUILD_ARTIFACT_NAME	infra-templates`
SERVICE_CONNECTION_NAME	<your_service_connection_name>
TF_VAR_elasticsearch_secrets_keyvault_name	osducommon<your_unique>-kv
TF_VAR_elasticsearch_secrets_keyvault_resource_group	osdu-common-<your_unique>
TF_VAR_remote_state_account	osducommon<your_unique>
TF_VAR_remote_state_container	remote-state-container

Setup and Configure the ADO Library Infrastructure Pipeline Variables - demo

Variable Value

ARM_SUBSCRIPTION_ID <your_subscription_id>

TF_VAR_resource_group_location centralus

TF_VAR_cosmosdb_replica_location eastus2

Variable	Value
ARM_SUBSCRIPTION_ID	<your_subscription_id>
TF_VAR_resource_group_location	centralus
TF_VAR_cosmosdb_replica_location	eastus2

You can specify the desired region locations you wish.

Setup and Configure the ADO Library Infrastructure Pipeline Secrets - demo

Variable Value

elastic-endpoint-ado-demo *********

elastic-username-ado-demo *********

elastic-password-ado-demo *********

Variable	Value
elastic-endpoint-ado-demo	`*********`
elastic-username-ado-demo	`*********`
elastic-password-ado-demo	`*********`

This should be linked Secrets from Azure Key Vault osducommon<your_unique>-kv

Setup 2 Secure Files
- azure-aks-gitops-ssh-key
- azure-aks-node-ssh-key.pub

** This is future AKS work but required. Ensure the names of files uploaded have the exact names listed which will require renaming the .ssh key information created by the script.

Add a Pipeline osdu-infrastructure --> azure-pipelines.yml and execute it.

Deploy OSDU Services

This typically takes about 3-4 hours to complete.

Setup Common ADO Libraries

Setup and Configure the ADO Library Azure Common

Variable	Value
AGENT_POOL	`Hosted Ubuntu 1604`
AZURE_AD_APP_RESOURCE_ID	`$(aad-client-id)`
AZURE_AD_OTHER_APP_RESOURCE_ID	`$(osdu-infra-<your_unique>-test-app-id)`
AZURE_DEPLOY_APPSERVICE_PLAN	`$(ENVIRONMENT_RG_PREFIX)-$(PREFIX_BASE)-sp`
AZURE_DEPLOY_CLIENT_ID	`$(app-dev-sp-username)`
AZURE_DEPLOY_CLIENT_SECRET	`$(app-dev-sp-password)`
AZURE_DEPLOY_RESOURCE_GROUP	`$(ENVIRONMENT_RG_PREFIX)-$(PREFIX_BASE)-app-rg`
AZURE_DEPLOY_TENANT	`$(app-dev-sp-tenant-id)`
AZURE_ENTITLEMENTS_SERVICE_NAME	`$(ENVIRONMENT_SERVICE_PREFIX)-entitlements`
AZURE_INDEXER_SERVICE_NAME	`$(ENVIRONMENT_SERVICE_PREFIX)-indexer`
AZURE_LEGAL_SERVICE_NAME	`$(ENVIRONMENT_SERVICE_PREFIX)-legal`
AZURE_LEGAL_SERVICEBUS	`$(sb-connection)`
AZURE_LEGAL_TOPICNAME	`legaltags`
AZURE_SEARCH_SERVICE_NAME	`$(ENVIRONMENT_SERVICE_PREFIX)-search`
AZURE_STORAGE_ACCOUNT	`$(ENVIRONMENT_STORAGE_PREFIX)sa`
AZURE_STORAGE_SERVICE_NAME	`$(ENVIRONMENT_SERVICE_PREFIX)-storage`
AZURE_TESTER_SERVICEPRINCIPAL_SECRET	`$(app-dev-sp-password)`
CONTAINER_REGISTRY_NAME	`$(ENVIRONMENT_STORAGE_PREFIX)cr`
DEPLOY_ENV	`empty`
DOMAIN	`contoso.com`
ENTITLEMENT_URL	`https://$(AZURE_ENTITLEMENTS_SERVICE_NAME).azurewebsites.net/`
EXPIRED_TOKEN
FUNCTION_APP_NAME	`$(ENVIRONMENT_BASE_NAME_21)-enque`
LEGAL_URL	`https://$(AZURE_LEGAL_SERVICE_NAME).azurewebsites.net/`
INTEGRATION_TESTER	`$(app-dev-sp-username)`
MY_TENANT	`opendes`
NO_DATA_ACCESS_TESTER	`$(osdu-infra-azg-test-app-noaccess-id)`
NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET	`$(osdu-infra-azg-test-app-noaccess-key)`
PREFIX_BASE	`osdu-r2`
PUBSUB_TOKEN	`az`
RESOURCE_GROUP_NAME	`$(ENVIRONMENT_RG_PREFIX)-$(PREFIX_BASE)-app-rg`
SEARCH_URL	`https://$(AZURE_SEARCH_SERVICE_NAME).azurewebsites.net/`
SERVICE_CONNECTION_NAME	`osdu-infrastructure`
STORAGE_URL	`https://$(AZURE_STORAGE_SERVICE_NAME).azurewebsites.net/`
GOOGLE_CLOUD_PROJECT	`opendes`

There is a current bug in legal that is pending being fixed the requires the GOOGLE_CLOUD_PROJECT variable to be set to build legal-core and pass the unit tests.

Setup and Configure the ADO Library Azure Common Secrets

This Library is linked to the Common Key Vault

osdu-infra-{unique}-test-app-id
osdu-infra-{unique}-test-app-key
osdu-infra-{unique}-test-app-noaccess-id
osdu-infra-{unique}-test-app-noaccess-key

Setup Environment ADO Libraries

Setup and Configure the ADO Library Azure Target Env - demo

Variable	Value
ENVIRONMENT_BASE_NAME_21	(ie: ado-demo-5xn-5xny9wvc)
ENVIRONMENT_RG_PREFIX	(ie: ado-demo-5xny9wvc)
ENVIRONMENT_SERVICE_PREFIX	$(ENVIRONMENT_BASE_NAME_21)-au
ENVIRONMENT_STORAGE_PREFIX	(ie: adodemo5xn5xny9wvc)
AZURE_DEPLOY_SUBSCRIPTION	<your_subscription_id>

Setup and Configure the ADO Library Azure Target Env Secrets - demo

This Library is linked to the Enviroment Key Vault

aad-client-id
app-dev-sp-password
app-dev-sp-username
appinsights-key
cosmos-connection
cosmos-endpoint
cosmos-primary-key
elastic-endpoint
elastic-password
elastic-username
entitlement-key
sb-connection
storage-account-key

Setup Service ADO Libraries

Setup and Configure the ADO Library Azure Service Release - entitlements

Variable	Value
MAVEN_DEPLOY_GOALS	`azure-webapp:deploy`
MAVEN_DEPLOY_OPTIONS	`--settings $(System.DefaultWorkingDirectory)/drop/maven/settings.xml -DAZURE_DEPLOY_TENANT=$(AZURE_DEPLOY_TENANT) -DAZURE_DEPLOY_CLIENT_ID=$(AZURE_DEPLOY_CLIENT_ID) -DAZURE_DEPLOY_CLIENT_SECRET=$(AZURE_DEPLOY_CLIENT_SECRET) -Dazure.appservice.resourcegroup=$(AZURE_DEPLOY_RESOURCE_GROUP) -Dazure.appservice.plan=$(AZURE_DEPLOY_APPSERVICE_PLAN) -Dazure.appservice.appname=$(AZURE_ENTITLEMENTS_SERVICE_NAME) -Dazure.appservice.subscription=$(AZURE_DEPLOY_SUBSCRIPTION)`
MAVEN_DEPLOY_POM_FILE_PATH	`drop`
MAVEN_INTEGRATION_TEST_OPTIONS	-DDOMAIN=$(DOMAIN) -DENTITLEMENT_URL=$(ENTITLEMENT_URL) -DENTITLEMENT_APP_KEY=$(entitlement-key) -DMY_TENANT=$(MY_TENANT) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_TENANT_ID=$(AZURE_DEPLOY_TENANT) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DAZURE_AD_OTHER_APP_RESOURCE_ID=$(AZURE_AD_OTHER_APP_RESOURCE_ID) -DEXPIRED_TOKEN=$(EXPIRED_TOKEN) -DENTITLEMENT_MEMBER_NAME_VALID=$(INTEGRATION_TESTER) -DENTITLEMENT_MEMBER_NAME_INVALID=InvalidTestAdmin -DENTITLEMENT_GROUP_NAME_VALID=integ.test.data.creator
MAVEN_INTEGRATION_TEST_POM_FILE_PATH	`drop/deploy/integration-tests`
SERVICE_RESOURCE_NAME	`$(AZURE_ENTITLEMENTS_SERVICE_NAME)`

Setup and Configure the ADO Library Azure Service Release - legal

Variable	Value
MAVEN_DEPLOY_GOALS	`azure-webapp:deploy`
MAVEN_DEPLOY_OPTIONS	`--settings $(System.DefaultWorkingDirectory)/drop/provider/legal-azure/maven/settings.xml -DAZURE_DEPLOY_TENANT=$(AZURE_DEPLOY_TENANT) -DAZURE_DEPLOY_CLIENT_ID=$(AZURE_DEPLOY_CLIENT_ID) -DAZURE_DEPLOY_CLIENT_SECRET=$(AZURE_DEPLOY_CLIENT_SECRET) -Dazure.appservice.resourcegroup=$(AZURE_DEPLOY_RESOURCE_GROUP) -Dazure.appservice.plan=$(AZURE_DEPLOY_APPSERVICE_PLAN) -Dazure.appservice.appname=$(AZURE_LEGAL_SERVICE_NAME) -Dazure.appservice.subscription=$(AZURE_DEPLOY_SUBSCRIPTION)`
MAVEN_DEPLOY_POM_FILE_PATH	`drop/provider/legal-azure`
MAVEN_INTEGRATION_TEST_OPTIONS	-DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DHOST_URL=$(LEGAL_URL) -DENTITLEMENT_URL=$(ENTITLEMENT_URL) -DMY_TENANT=$(MY_TENANT) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_TENANT_ID=$(AZURE_DEPLOY_TENANT) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DAZURE_LEGAL_STORAGE_ACCOUNT=$(AZURE_STORAGE_ACCOUNT) -DAZURE_LEGAL_STORAGE_KEY=$(storage-account-key) -DAZURE_LEGAL_SERVICEBUS=$(AZURE_LEGAL_SERVICEBUS) -DAZURE_LEGAL_TOPICNAME=$(AZURE_LEGAL_TOPICNAME)
MAVEN_INTEGRATION_TEST_POM_FILE_PATH	`drop/deploy/testing/legal-test-azure/pom.xml`
SERVICE_RESOURCE_NAME	`$(AZURE_LEGAL_SERVICE_NAME)`

- Setup and Configure the ADO Library Azure Service Release - storage

Variable	Value
MAVEN_DEPLOY_GOALS	`azure-webapp:deploy`
MAVEN_DEPLOY_OPTIONS	`--settings $(System.DefaultWorkingDirectory)/drop/provider/storage-azure/maven/settings.xml -DAZURE_DEPLOY_TENANT=$(AZURE_DEPLOY_TENANT) -DAZURE_DEPLOY_CLIENT_ID=$(AZURE_DEPLOY_CLIENT_ID) -DAZURE_DEPLOY_CLIENT_SECRET=$(AZURE_DEPLOY_CLIENT_SECRET) -Dazure.appservice.resourcegroup=$(AZURE_DEPLOY_RESOURCE_GROUP) -Dazure.appservice.plan=$(AZURE_DEPLOY_APPSERVICE_PLAN) -Dazure.appservice.appname=$(AZURE_STORAGE_SERVICE_NAME) -Dazure.appservice.subscription=$(AZURE_DEPLOY_SUBSCRIPTION)`
MAVEN_DEPLOY_POM_FILE_PATH	`drop/provider/storage-azure`
MAVEN_INTEGRATION_TEST_OPTIONS	-DDOMAIN=$(DOMAIN) -DSTORAGE_URL=$(STORAGE_URL) -DLEGAL_URL=$(LEGAL_URL) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DNO_DATA_ACCESS_TESTER=$(NO_DATA_ACCESS_TESTER) -DTESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DNO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET=$(NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_TENANT_ID=$(AZURE_DEPLOY_TENANT) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DDEPLOY_ENV=$(DEPLOY_ENV) -DPUBSUB_TOKEN=$(PUBSUB_TOKEN) -DTENANT_NAME=$(MY_TENANT) -DAZURE_STORAGE_ACCOUNT=$(AZURE_STORAGE_ACCOUNT)
MAVEN_INTEGRATION_TEST_POM_FILE_PATH	`drop/deploy/testing/storage-test-azure`
SERVICE_RESOURCE_NAME	`$(AZURE_STORAGE_SERVICE_NAME)`
`

- Setup and Configure the ADO Library Azure Service Release - indexer

Variable	Value
MAVEN_DEPLOY_GOALS	`azure-webapp:deploy`
MAVEN_DEPLOY_OPTIONS	--settings $(System.DefaultWorkingDirectory)/drop/provider/indexer-azure/maven/settings.xml -DAZURE_DEPLOY_TENANT=$(AZURE_DEPLOY_TENANT) -DAZURE_DEPLOY_CLIENT_ID=$(AZURE_DEPLOY_CLIENT_ID) -DAZURE_DEPLOY_CLIENT_SECRET=$(AZURE_DEPLOY_CLIENT_SECRET) -Dazure.appservice.resourcegroup=$(AZURE_DEPLOY_RESOURCE_GROUP) -Dazure.appservice.plan=$(AZURE_DEPLOY_APPSERVICE_PLAN) -Dazure.appservice.appname=$(AZURE_INDEXER_SERVICE_NAME) -Dazure.appservice.subscription=$(AZURE_DEPLOY_SUBSCRIPTION) -DELASTIC_USER_NAME=$(elastic-username) -DELASTIC_PASSWORD=$(elastic-password) -DELASTIC_HOST=$(elastic-host) -DELASTIC_PORT=$(elastic-port)
MAVEN_DEPLOY_POM_FILE_PATH	`drop/provider/indexer-azure`
MAVEN_INTEGRATION_TEST_OPTIONS	-DAZURE_AD_TENANT_ID=$(AZURE_DEPLOY_TENANT) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DELASTIC_USER_NAME=$(elastic-username) -DELASTIC_PASSWORD=$(elastic-password) -DELASTIC_HOST=$(elastic-host) -DELASTIC_PORT=$(elastic-port) -DDEFAULT_DATA_PARTITION_ID_TENANT1=$(MY_TENANT) -DDEFAULT_DATA_PARTITION_ID_TENANT2=othertenant2 -DLEGAL_TAG=opendes-public-usa-dataset-7643990 -DOTHER_RELEVANT_DATA_COUNTRIES=US -DENTITLEMENTS_DOMAIN=contoso.com -DENVIRONMENT=CLOUD -DSTORAGE_HOST=$(STORAGE_URL)
MAVEN_INTEGRATION_TEST_POM_FILE_PATH	`drop/deploy/testing/indexer-test-azure`
SERVICE_RESOURCE_NAME	`$(AZURE_INDEXER_SERVICE_NAME)`

- Setup and Configure the ADO Library Azure Service Release - search

Variable	Value
MAVEN_DEPLOY_GOALS	`azure-webapp:deploy`
MAVEN_DEPLOY_OPTIONS	`--settings $(System.DefaultWorkingDirectory)/drop/provider/search-azure/maven/settings.xml -DAZURE_DEPLOY_TENANT=$(AZURE_DEPLOY_TENANT) -DAZURE_DEPLOY_CLIENT_ID=$(AZURE_DEPLOY_CLIENT_ID) -DAZURE_DEPLOY_CLIENT_SECRET=$(AZURE_DEPLOY_CLIENT_SECRET) -Dazure.appservice.resourcegroup=$(AZURE_DEPLOY_RESOURCE_GROUP) -Dazure.appservice.plan=$(AZURE_DEPLOY_APPSERVICE_PLAN) -Dazure.appservice.appname=$(AZURE_SEARCH_SERVICE_NAME) -Dazure.appservice.subscription=$(AZURE_DEPLOY_SUBSCRIPTION)`
MAVEN_DEPLOY_POM_FILE_PATH	`drop/provider/search-azure`
MAVEN_INTEGRATION_TEST_OPTIONS	-DOTHER_RELEVANT_DATA_COUNTRIES= -DINTEGRATION_TEST_AUDIENCE= -DDEFAULT_DATA_PARTITION_ID_TENANT1=$(MY_TENANT) -DDEFAULT_DATA_PARTITION_ID_TENANT2=othertenant2 -DELASTIC_USER_NAME=$(elastic-username) -DELASTIC_PASSWORD=$(elastic-password) -DELASTIC_HOST=$(elastic-host) -DELASTIC_PORT=$(elastic-port) -DINDEXER_HOST=$() -DENTITLEMENTS_DOMAIN=$(DOMAIN) -DSEARCH_HOST=$(SEARCH_URL)api/search/v2/ -DSTORAGE_HOST=$() -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_TENANT_ID=$(AZURE_DEPLOY_TENANT) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID)
MAVEN_INTEGRATION_TEST_POM_FILE_PATH	`drop/deploy/testing/integration-tests/search-test-azure`
SERVICE_RESOURCE_NAME	`$(AZURE_SEARCH_SERVICE_NAME)`

Load Cosmos DB Integration Test Data

The data to be loaded before services are deployed and can be found in the osdu-infrastructure repository osdu-infrastructure/docs/osdu/integration-test-data/ and has to be modified with environment specific information as necessary.

tenant_info_1.json
tenant_info_2.json
user_info_1.json
user_info_2.json
legal_tag_1.json
legal_tag_2.json
legal_tag_3.json
storage_schema_1.json
storage_schema_2.json
storage_schema_3.json
storage_schema_4.json
storage_schema_5.json
storage_schema_6.json
storage_schema_7.json
storage_schema_8.json
storage_schema_9.json
storage_schema_10.json
storage_schema_11.json

Configure the ADO Service Pipelines

Create the pipelines and run things in this exact order.

Add a Pipeline entitlements-azure --> Repo: entitlements-azure Path:/devops/azure-pipelines.yml and execute it.
- This pipeline will have to be run twice for integration tests to pass due to a preload data issue.
Add a Pipeline legal --> Repo: legal Path:/devops/azure-pipelines.yml and execute it.
Add a Pipeline indexer-queue --> Repo: indexer-queue Path:/devops/azure-pipelines.yml and execute it.

Note: This is a manual deploy at the moment.
Add a Pipeline storage --> Repo: storage Path:/devops/azure-pipelines.yml and execute it.

Note: The integration tests will fail due to delivery being part of storage which requires indexer and search. Ignore the error and proceed.
Add a Pipeline indexer --> Repo: indexer Path:/devops/azure-pipelines.yml and execute it.
Add a Pipeline search --> Repo: search Path:/devops/azure-pipelines.yml and execute it.
Rerun the Pipeline for storage so that tests now pass.

danielscholl/validation.md