How to run Enterprise Search with Kibana in ECK

This guide describes the process of configuring Enterprise Search and Kibana running in Elastic Cloud on Kubernetes (ECK) to work together. The final setup uses end-to-end TLS encryption with self-signed certificates managed by ECK.

Prerequisites

Before you begin, you need a few pieces in place:

Get a Kubernetes cluster
Install ECK on it
Install Elasticsearch using ECK (do not install Kibana just yet)

You are now ready to continue with Kibana and Enterprise Search configuration.

Setting up Enterprise Search

External URL

Before you continue with the Enterprise Search deployment, there is one thing you need to decide: how will end-users reach your Enterprise Search instance after it is deployed. Depending on this decision, you will need to configure Enterprise Search deployment accordingly so that it knows where it is deployed. To do this, you will use the ent_search.external_url configuration file option, setting it to the URL of your deployment as users will see it.

For this guide, we'll assume you'll be using the deployment internally within the Kubernetes cluster, so you will use the internal name Kubernetes gives to your deployment automatically. If your situation is different, you need to make sure the following is true:

Your deployment is reachable via a name that is resolvable from within the Kubernetes cluster
Your deployment is configured with a valid SSL certificate that matches the name you are using

Provisioning an Enterprise Deployment

Following the quickstart guide, let's deploy an Enterprise Search instance:

cat <<EOF | kubectl apply -f -
apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-quickstart
spec:
  version: 7.13.0
  count: 1
  elasticsearchRef:
    name: quickstart
  config:
    ent_search:
      external_url: https://enterprise-search-quickstart-ent-http:3002
EOF

Please note: The ent_search.external_url setting in this case matches the name of the EnterpriseSearch object and is the name ECK will automatically generate for the deployment with a self-signed SSL certificate. If you are using a proxy in front of Enterprise Search, you'll need to make sure the instance is configured with the correct certificate, etc and that the name used to reach the proxy is specified in ent_search.external_url. More information on using custom SSL certificates could be found in ECK documentation.

After this is done, you can follow the quickstart guide to check the status of the deployment, etc. After you have done that and you are sure the deployment works as expected, you can continue to setting up Kibana to work with this deployment.

Setting up Kibana

Before we continue, you need to know how Kibana deployments interact with Enterprise Search. There are two separate channels of interaction between Kibana and Enterprise Search deployments:

When you open an Enterprise Search page in Kibana, the code running in your browser will talk to Kibana server running in Kubernetes and will ask it to call Enterprise Search.
At some point, you will click something on the Enterprise Search page in Kibana and your browser may be sent directly to the dedicated Enterprise Search UI (this goes away in 8.0 and is getting deprecated starting with 7.14).

The latter type of interaction should already work for you (since you have tested direct access to Enterprise Search in the previous section of this guide). This section of the guide will be primarily concerned with making sure Kibana server can interact with your Enterprise Search deployment.

Few things need to be true for Kibana interactions with Enterprise Search to work:

Kibana config file needs to be updated with an enterpriseSearch.host setting, pointing at your Enterprise Search deployment.
Kibana needs to be able to resolve the name specified in enterpriseSearch.host.
Kibana needs to trust the SSL certificate presented to it during the connection to enterpriseSearch.host.

Let's figure out how to do each of those separately.

Specifying Enterprise Search endpoint

To let Kibana know where Enterprise Search deployment is and how to talk to it, we need to add an enterpriseSearch.host setting to its configuration file. In ECK world, this is done via the config section of the Kibana resource definition.

Here is how it may look (this is not the final configuration, just an example of how to use the config setting):

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.13.0
  count: 1
  elasticsearchRef:
    name: quickstart
  config:
    enterpriseSearch.host: https://your-ent-search.deployment.host:3002

What DNS name to use for Enterprise Search endpoint

The host specified in the enterpriseSearch.host setting for Kibana needs to be reachable from Kibana deployment and it should be supported by the SSL certificate deployed on the Enterprise Search instance. One way to achieve that may be to use the same host value listed in ent_search.external_url setting in Enterprise Search. If you use a different name, always make sure the SSL certificate name supports it (this is true by default in ECK).

Since we deploy our solution in ECK, each product will have a Kubernetes service automatically configured for it and DNS will be automatically set up within the cluster to make things work for us. In our quickstart case, the Enterprise Search deployment created above has a name enterprise-search-quickstart-ent-http.default.svc.cluster.local based on the standard Kubernetes naming scheme. Another way to refer to the same DND name is simply enterprise-search-quickstart-ent-http (and auto-generated SSL certs in ECK use this short version, so we'll use it as well).

Here is how it may look (again, not the final version yet):

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.13.0
  count: 1
  elasticsearchRef:
    name: quickstart
  config:
    # Let Kibana know where Enterprise Search is deployed
    enterpriseSearch.host: https://enterprise-search-quickstart-ent-http:3002

Trusting SSL Certificates

All components within our clusters managed by ECK have SSL configured for us automatically. This is great, because we don't need to worry about certificate management, but it presents us with a challenge when we try to connect to those services from places that do not know how to properly verify SSL certificates used within our deployments.

One of the places that need some adjustments in our setup is Kibana to Enterprise Search server communications. By default, Kibana has no idea how to verify self-signed SSL certificates presented by Enterprise Search and will fail requests to it with the following error:

{
  "type": "log",
  "@timestamp": "2021-05-26T12:28:36Z",
  "tags": [
    "error",
    "plugins",
    "enterpriseSearch"
  ],
  "pid": 6,
  "message": "Cannot connect to App Search: FetchError: request to https://enterprise-search-quickstart-ent-http:3002/as/engines/collection?type=indexed&page%5Bcurrent%5D=1&page%5Bsize%5D=10 failed, reason: self signed certificate in certificate chain"
}

Notice the reason: self signed certificate in certificate chain part of the error. That is something we need to fix.

To fix the problem with self-signed certificates, we need let Kibana know where to get the custom Certificate Authority certificate we have used to sign our Enterprise Search SSL certificate. This is done by setting a config file option called enterpriseSearch.ssl.certificateAuthorities and pointing it at a file containing the custom CA certificate. Luckily for us, ECK stores this certificate in a Kubernetes secret available for us to use.

Here is how we can let ECK know we want to make the custom CA certificate available to Kibana:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.13.0
  count: 1
  elasticsearchRef:
    name: quickstart

  # Customize Kubernetes pods used for Kibana deployment
  podTemplate:
    spec:
      volumes:
        # Take Enterprise Search certificate information and make it available as a disk volume
        - name: elastic-internal-enterprise-search-http-certificates
          secret:
            defaultMode: 420
            optional: false
            secretName: enterprise-search-quickstart-ent-http-certs-internal

      containers:
      - name: kibana

        # Mount Enterprise Search certificate information volume as a directory into Kibana containers
        volumeMounts:
          - mountPath: /mnt/elastic-internal/enterprise-search-certs
            name: elastic-internal-enterprise-search-http-certificates
            readOnly: true

  config:
    # Let Kibana know where Enterprise Search is deployed
    enterpriseSearch.host: https://enterprise-search-quickstart-ent-http:3002

    # Let Kibana know where to get the custom CA certificate
    enterpriseSearch.ssl.certificateAuthorities: /mnt/elastic-internal/enterprise-search-certs/ca.crt

In this configuration we do a few things:

We take the Kubernetes secret holding SSL certificate information for Enterprise Search and we make it available in Kibana containers as a disk volume called elastic-internal-enterprise-search-http-certificates.
We mount the volume into Kibana containers as a directory called /mnt/elastic-internal/enterprise-search-certs.
Finally, we let Kibana know that our custom CA certificate is available as a file called /mnt/elastic-internal/enterprise-search-certs/ca.crt.

After all of this is done, we can use kubectl apply to create a Kibana deployment that will know how to connect to Enterprise Search and will be able to validate its SSL certificate:

cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.13.0
  count: 1
  elasticsearchRef:
    name: quickstart

  # Customize Kubernetes pods used for Kibana deployment
  podTemplate:
    spec:
      volumes:
        # Take Enterprise Search certificate information and make it available as a disk volume
        - name: elastic-internal-enterprise-search-http-certificates
          secret:
            defaultMode: 420
            optional: false
            secretName: enterprise-search-quickstart-ent-http-certs-internal

      containers:
      - name: kibana

        # Mount Enterprise Search certificate information volume as a directory into Kibana containers
        volumeMounts:
          - mountPath: /mnt/elastic-internal/enterprise-search-certs
            name: elastic-internal-enterprise-search-http-certificates
            readOnly: true

  config:
      # Let Kibana know where Enterprise Search is deployed
      enterpriseSearch.host: https://enterprise-search-quickstart-ent-http:3002

      # Let Kibana know where to get the custom CA certificate
      enterpriseSearch.ssl.certificateAuthorities: /mnt/elastic-internal/enterprise-search-certs/ca.crt

EOF

From this point forward, you can follow the standard Kibana quickstart guide for more information on working with a Kibana deployment on ECK.

kovyrin/ent-search-and-kibana-on-eck.md