Islandora Isle-DC Service Discovery and Network Check Troubleshooting Script

README.md

Islandora Isle-DC Service Discovery and Network Check Script (DRAFT)

This script is designed to verify the network configuration and DNS resolution of services running in Docker containers specifically within an Islandora Isle-DC environment. It helps ensure that all microservices and components in the Isle-DC stack are correctly connected and communicating.

Features

• Service Discovery: Automatically identifies services defined in your docker-compose.yml file for Isle-DC.
• Network Validation: Checks if each Isle-DC service is connected to the correct Docker network.
• Ping Tests: Performs ping tests between all running Isle-DC services to verify inter-container communication.
• DNS Resolution: Ensures that DNS is properly configured and resolves between services in the Isle-DC environment.
• Summary Report: Provides a detailed summary of successful checks and any errors encountered.

Requirements

• Docker and Docker Compose installed.

• yq installed for parsing YAML files. You can install yq using the following command:

  sudo apt-get install yq

  Alternatively, you can install yq using snap:

  sudo snap install yq

  Installation on macOS:

  brew install yq

  Installation on Windows:

  choco install yq

How to Use

1. Download the Script:

   You can download the script directly from this Gist by clicking on the "Download ZIP" button or by saving the check_service_discovery.sh file manually. Or you can clone this gist

   git clone https://gist.github.com/DonRichards/cf3334218abdd23601564d600f55e94a.git docker_compose_network_check
   mv docker_compose_network_check/check_service_discovery.sh .

2. Place the Script in Your Isle-DC Project: Ensure that the script is located in the same directory as your Isle-DC docker-compose.yml file.

3. Make the Script Executable:

   chmod +x check_service_discovery.sh

4. Run the Script:

   ./check_service_discovery.sh

Output

The script will perform the following checks:

• Service Discovery: Identifies services defined in the Isle-DC docker-compose.yml file and checks if their corresponding containers are running.
• Network Check: Verifies that each Isle-DC service is connected to the expected Docker network.
• Ping Tests: Ensures that each container can communicate with the other containers in the Isle-DC stack.
• DNS Resolution: Verifies that DNS resolution is configured and working between containers.

At the end, the script provides a summary of the results:

• The number of successful checks.
• Any errors encountered during the checks (e.g., missing containers, network configuration issues).

Sample Output

Starting network and DNS configuration checks...
Extracting services from docker-compose.yml...
Raw extracted services:
activemq
alpaca
...
Filtered valid services:
activemq
alpaca
...
Detected Services: activemq alpaca blazegraph cantaloupe crayfits drupal fcrepo fits homarus houdini hypercube mariadb milliner solr traefik
Mapped Containers: isle-dc-activemq-1 isle-dc-alpaca-1 isle-dc-blazegraph-1 isle-dc-cantaloupe-1 isle-dc-crayfits-1 isle-dc-drupal-1 isle-dc-fcrepo-1 isle-dc-fits-1 isle-dc-homarus-1 isle-dc-houdini-1 isle-dc-hypercube-1 isle-dc-mariadb-1 isle-dc-milliner-1 isle-dc-solr-1 traefik
Checking network for container: isle-dc-activemq-1
  isle-dc-activemq-1 is connected to network (isle_dc_default).
Checking network for container: traefik
  traefik is connected to network (isle_dc_default).

===== SUMMARY =====
Successful checks: 435
No errors detected. All checks passed successfully.
===================

Troubleshooting

• Missing or Misconfigured Services: If the script detects that a service is missing or misconfigured, check your Isle-DC docker-compose.yml and ensure all services are properly defined and running.
• Network Issues: If services are not connected to the correct network, ensure that Docker networks are properly configured and that containers are using the correct network bridge.

If you find something that should be added to this message me in Islandora's Slack Channel.

check_service_discovery.sh

#!/bin/bash

# Initialize summary variables
ERRORS=()
SUCCESSFUL_CHECKS=0

# Function to extract the correct Docker network name for each container
get_container_network() {
  NETWORK_NAME=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' "$1")
  if [ -z "$NETWORK_NAME" ]; then
    NETWORK_NAME="unknown"
  fi
  echo "$NETWORK_NAME"
}

# Function to extract the service names from the docker-compose.yml file
get_services() {
  echo "Extracting services from docker-compose.yml..."

  # Extract service names from the 'services' section of the docker-compose.yml file
  SERVICES=($(yq e '.services | keys | .[]' docker-compose.yml))

  # Debug output: list all extracted services
  echo "Raw extracted services:"
  printf "%s\n" "${SERVICES[@]}"

  # Filter services that match running containers
  VALID_SERVICES=($(docker ps --format '{{.Names}}' | sed 's/^isle-dc-\(.*\)-1$/\1/' | grep -v '^traefik$'))

  # Add 'traefik' as a valid service
  if docker ps --format '{{.Names}}' | grep -q '^traefik$'; then
    VALID_SERVICES+=("traefik")
  fi

  FILTERED_SERVICES=()
  for SERVICE in "${SERVICES[@]}"; do
    if [[ " ${VALID_SERVICES[@]} " =~ " ${SERVICE} " ]]; then
      FILTERED_SERVICES+=("$SERVICE")
    fi
  done

  # Debug output: list valid services after filtering
  echo "Filtered valid services:"
  printf "%s\n" "${FILTERED_SERVICES[@]}"

  if [ ${#FILTERED_SERVICES[@]} -eq 0 ]; then
    echo "No valid services found in docker-compose.yml. Please check the file format."
  else
    echo "Detected Services: ${FILTERED_SERVICES[*]}"
  fi

  SERVICES=("${FILTERED_SERVICES[@]}")
}

# Function to map service names to container names
map_services_to_containers() {
  CONTAINERS=()
  for SERVICE in "${SERVICES[@]}"; do
    if [ "$SERVICE" == "traefik" ]; then
      CONTAINER_NAME="traefik"
    else
      CONTAINER_NAME="isle-dc-${SERVICE}-1"
    fi
    CONTAINERS+=("$CONTAINER_NAME")
  done
  echo "Mapped Containers: ${CONTAINERS[*]}"
}

# Function to check if each container is connected to its correct network
check_network() {
  for CONTAINER in "${CONTAINERS[@]}"; do
    if docker inspect "$CONTAINER" > /dev/null 2>&1; then
      echo "Checking network for container: $CONTAINER"
      CONTAINER_NETWORK=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' $CONTAINER)
      if [ -n "$CONTAINER_NETWORK" ]; then
        echo "  $CONTAINER is connected to network ($CONTAINER_NETWORK)."
        SUCCESSFUL_CHECKS=$((SUCCESSFUL_CHECKS + 1))
      else
        echo "  ERROR: $CONTAINER is NOT connected to any network."
        ERRORS+=("No network found for $CONTAINER")
      fi
    else
      echo "Error: No such container: $CONTAINER"
      ERRORS+=("No such container: $CONTAINER")
    fi
  done
}

# Function to ping each container from all other containers
ping_services() {
  for SOURCE in "${CONTAINERS[@]}"; do
    if docker inspect "$SOURCE" > /dev/null 2>&1; then
      echo "Pinging services from: $SOURCE"
      for TARGET in "${CONTAINERS[@]}"; do
        if [ "$SOURCE" != "$TARGET" ] && docker inspect "$TARGET" > /dev/null 2>&1; then
          echo "  Pinging $TARGET from $SOURCE..."
          docker exec "$SOURCE" ping -c 4 "$TARGET"
          SUCCESSFUL_CHECKS=$((SUCCESSFUL_CHECKS + 1))
        fi
      done
    fi
  done
}

# Function to check DNS resolution inside each container
check_dns_resolution() {
  for CONTAINER in "${CONTAINERS[@]}"; do
    if docker inspect "$CONTAINER" > /dev/null 2>&1; then
      echo "Checking DNS resolution for container: $CONTAINER"
      docker exec "$CONTAINER" cat /etc/resolv.conf
      echo "  Resolving DNS names from $CONTAINER..."
      for TARGET in "${CONTAINERS[@]}"; do
        if [ "$CONTAINER" != "$TARGET" ] && docker inspect "$TARGET" > /dev/null 2>&1; then
          echo "    Resolving $TARGET..."
          docker exec "$CONTAINER" getent hosts "$TARGET"
          SUCCESSFUL_CHECKS=$((SUCCESSFUL_CHECKS + 1))
        fi
      done
    fi
  done
}

# Function to summarize findings
summarize_findings() {
  echo
  echo "===== SUMMARY ====="
  echo "Successful checks: $SUCCESSFUL_CHECKS"
  if [ ${#ERRORS[@]} -gt 0 ]; then
    echo "Errors detected:"
    for ERROR in "${ERRORS[@]}"; do
      echo "  - $ERROR"
    done
  else
    echo "No errors detected. All checks passed successfully."
  fi
  echo "==================="
}

# Run the checks
echo "Starting network and DNS configuration checks..."
get_services
map_services_to_containers
check_network
ping_services
check_dns_resolution

# Summarize findings
summarize_findings

echo "All checks completed."