debug-server.md

How to Set Up Debug Servers in Production

In the world of software development, the ability to debug issues in a production-like environment is crucial. However, routing debug traffic to specific servers without affecting regular users can be challenging. This article explores various methods to set up debug servers in production environments, focusing on Kubernetes and cloud solutions.

Kubernetes Solutions

Kubernetes offers several ways to route debug traffic to specific pods. We'll explore three methods: using Ingress, special domains, and HTTP headers.

1. Using Ingress

Kubernetes Ingress allows you to route traffic based on rules. You can use it to direct debug traffic to specific debug pods.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-production
            port: 
              number: 80
  - host: debug.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-debug
            port: 
              number: 80

In this setup, regular traffic to example.com goes to the production service, while traffic to debug.example.com is routed to the debug service.

2. Special Domain

As shown in the Ingress example above, you can use a special subdomain (like debug.example.com) to route traffic to debug pods. This method is straightforward but requires additional DNS configuration.

3. HTTP Headers

You can use HTTP headers to route traffic to debug pods without changing the URL. This requires an Ingress controller that supports header-based routing, such as NGINX Ingress Controller.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($http_x_debug = "true") {
        set $proxy_upstream_name "myapp-debug";
      }
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-production
            port: 
              number: 80

With this configuration, requests with the header X-Debug: true will be routed to the debug service.

Cloud Solutions

Cloud providers offer their own solutions for routing debug traffic. Let's look at global load balancers and AWS-specific solutions.

Global Load Balancer

Many cloud providers offer global load balancers that can route traffic based on various criteria, including HTTP headers or URL paths. Here's a generic example of how you might configure a global load balancer:

loadBalancer:
  name: my-global-lb
  rules:
    - name: production-rule
      priority: 1
      conditions:
        - field: path-pattern
          values: ['/*']
      actions:
        - type: forward
          targetGroup: production-target-group
    - name: debug-rule
      priority: 2
      conditions:
        - field: http-header
          httpHeaderConfig:
            httpHeaderName: X-Debug
            values: ['true']
      actions:
        - type: forward
          targetGroup: debug-target-group

In this configuration, requests with the X-Debug: true header are sent to the debug target group, while all other requests go to the production target group.

AWS Solution

AWS offers the Application Load Balancer (ALB) which can route traffic based on rules, including HTTP headers. Here's an example using AWS CloudFormation:

Resources:
  MyALB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Scheme: internet-facing
      SecurityGroups: 
        - !Ref ALBSecurityGroup
      Subnets: 
        - !Ref PublicSubnet1
        - !Ref PublicSubnet2

  ALBListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref ProductionTargetGroup
      LoadBalancerArn: !Ref MyALB
      Port: 80
      Protocol: HTTP

  DebugListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref DebugTargetGroup
      Conditions:
        - Field: http-header
          HttpHeaderConfig:
            HttpHeaderName: X-Debug
            Values:
              - "true"
      ListenerArn: !Ref ALBListener
      Priority: 1

  ProductionTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckPath: /healthz
      Name: production-tg
      Port: 80
      Protocol: HTTP
      TargetType: ip
      VpcId: !Ref VPC

  DebugTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckPath: /healthz
      Name: debug-tg
      Port: 80
      Protocol: HTTP
      TargetType: ip
      VpcId: !Ref VPC

This AWS ALB setup routes traffic with the X-Debug: true header to the debug target group, while all other traffic goes to the production target group.

Conclusion

Setting up debug servers in production environments can greatly enhance your ability to troubleshoot issues. Whether you're using Kubernetes or cloud provider solutions, you have multiple options for routing debug traffic. The choice between using Ingress rules, special domains, HTTP headers, or cloud-specific load balancers depends on your specific infrastructure and requirements. By implementing these solutions, you can safely debug production issues without impacting your regular users.