Installing Mirantis k0rdent Enterprise Observability and FinOps#

Prerequisites#

Before beginning KOF installation, you should have the following components in place:

A Mirantis k0rdent Enterprise management cluster - You can get instructions to create one in the quickstart guide.
You will also need your infrastructure provider credentials, such as those shown in the guide for AWS.
- Note that you should skip the "Create your ClusterDeployment" and later sections.
Finally, select one of the options:
- DNS auto-config to automate the process for all regional clusters.
- Manual DNS config is applied later for each regional cluster separately and manually.
- Istio service mesh for secure connectivity between clusters. This is the only option which does not need access to create external DNS records for service endpoints such as kof.example.com.

Image Registry#

Let's configure the registry where the Docker images will be pulled from.

Create the global-values.yaml file:

global:
  registry: registry.mirantis.com/k0rdent-enterprise
  imageRegistry: registry.mirantis.com/k0rdent-enterprise
  image:
    registry: registry.mirantis.com/k0rdent-enterprise
  hub: registry.mirantis.com/k0rdent-enterprise/istio
  helmChartsRepo: oci://registry.mirantis.com/k0rdent-enterprise/charts
cert-manager-istio-csr:
  image:
    repository: registry.mirantis.com/k0rdent-enterprise/jetstack/cert-manager-istio-csr
cert-manager-service-template:
  skipVerifyJob: true
  repo:
    type: oci
    url: oci://registry.mirantis.com/k0rdent-enterprise/charts
cluster-api-visualizer:
  image:
    repository: registry.mirantis.com/k0rdent-enterprise/k0rdent
    tag: 1.4.1
external-dns:
  image:
    repository: registry.mirantis.com/k0rdent-enterprise/external-dns/external-dns
grafana-operator:
  image:
    repository: registry.mirantis.com/k0rdent-enterprise/grafana/grafana-operator
ingress-nginx-service-template:
  skipVerifyJob: true
  repo:
    type: oci
    url: oci://registry.mirantis.com/k0rdent-enterprise/charts
jaeger-operator:
  image:
    repository: registry.mirantis.com/k0rdent-enterprise/jaegertracing/jaeger-operator
kcm:
  kof:
    repo:
      spec:
        url: oci://registry.mirantis.com/k0rdent-enterprise/charts
opencost:
  opencost:
    exporter:
      image:
        registry: registry.mirantis.com/k0rdent-enterprise
    ui:
      image:
        registry: registry.mirantis.com/k0rdent-enterprise
opentelemetry-operator:
  manager:
    image:
      repository: registry.mirantis.com/k0rdent-enterprise/opentelemetry-operator/opentelemetry-operator
    collectorImage:
      repository: registry.mirantis.com/k0rdent-enterprise/otel/opentelemetry-collector-contrib
  kubeRBACProxy:
    image:
      repository: registry.mirantis.com/k0rdent-enterprise/brancz/kube-rbac-proxy

This file will be used in the next sections.

Air Gap#

For an air-gapped environment please apply additional steps below:

Replace all mentions of the registry.mirantis.com/k0rdent-enterprise in the global-values.yaml file and other examples with your own registry, for example, registry.local

Notice

Custom registry using HTTP redirect is not supported yet due to certificate verification issue. We plan to fix it in the next release.

Update global-values.yaml file using the secrets created on the Configuring a Custom OCI Registry for KCM components step. For example:

kcm:
  kof:
    repo:
      spec:
        secretRef:
          name: my-private-oci-registry-creds
        certSecretRef:
          name: registry-cert
cert-manager-service-template:
  repo:
    secretRef:
      name: my-private-oci-registry-creds
    certSecretRef:
      name: registry-cert
ingress-nginx-service-template:
  repo:
    secretRef:
      name: my-private-oci-registry-creds
    certSecretRef:
      name: registry-cert

Either apply the Istio section or one of the options below, required for DNS auto-config and Manual DNS config cases only.

To use self-signed certificates:

Add to global-values.yaml file:

storage:
  cert-manager:
    cluster-issuer:
      provider: self-signed

On step 9 in the Regional Cluster section apply:

metadata:
  annotations:
    k0rdent.mirantis.com/kof-http-config: '{"tls_config": {"insecure_skip_verify": true}}'

On step 7 in the Child Cluster section you may apply:

spec:
  config:
    clusterAnnotations:
      k0rdent.mirantis.com/kof-collectors-values: |
        kof:
          logs:
            tls_options:
              insecure_skip_verify: true
          metrics:
            tls_options:
              insecure_skip_verify: true
          traces:
            tls_options:
              insecure_skip_verify: true

Note: insecure_skip_verify is a temporary workaround auto-enabled by default until we implement a chain of trust in the next releases.

On step 1 in the Storing KOF data - From Management to Regional section you may apply:

cat >collectors-values.yaml <<EOF
kof:
  logs:
    endpoint: https://vmauth.$REGIONAL_DOMAIN/vli/insert/opentelemetry/v1/logs
    tls_options:
      insecure_skip_verify: true
  metrics:
    endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/insert/0/prometheus/api/v1/write
    tls_options:
      insecure_skip_verify: true
  traces:
    endpoint: https://jaeger.$REGIONAL_DOMAIN/collector
    tls_options:
      insecure_skip_verify: true
opencost:
  opencost:
    prometheus:
      external:
        url: https://vmauth.$REGIONAL_DOMAIN/vm/select/0/prometheus
EOF

Note: insecure_skip_verify is a temporary workaround auto-enabled by default until we implement a chain of trust in the next releases.

Note: opencost.prometheus.external.url and related values of the opencost chart do not support insecure_skip_verify for now.

To use your own cert-manager, add to global-values.yaml file:
```
storage:
  cert-manager:
    enabled: false
```
To use your own certificates:
- Wait until you apply the Regional Cluster section and get the regional-kubeconfig in the Verifying step.
- Create the next secrets there in kof namespace as documented in Ingress - TLS:
  - vmauth-tls
  - jaeger-ingress-tls-secret
  - grafana-cluster-tls
  - dex-ingress-tls-secret
- Or you can create these secrets in the management cluster and auto-distribute them to regional clusters using the clusterProfiles as documented in step 4 of the Management Cluster section.

DNS auto-config#

To avoid manual configuration of DNS records for service endpoints later, you can automate the process now using external-dns.

AWS#

For AWS in production, use the Node IAM Role or IRSA methods in production.

Just for the sake of this demo based on the aws-standalone template, however, you can use the most straightforward (though less secure) static credentials method:

Create an external-dns IAM user with this policy.

Create an access key and external-dns-aws-credentials file, as in:

[default]
aws_access_key_id = <EXAMPLE_ACCESS_KEY_ID>
aws_secret_access_key = <EXAMPLE_SECRET_ACCESS_KEY>

Create the external-dns-aws-credentials secret in the kof namespace:

kubectl create namespace kof
kubectl create secret generic \
  -n kof external-dns-aws-credentials \
  --from-file external-dns-aws-credentials

Azure#

To enable DNS auto-config on Azure, use DNS Zone Contributor.

Create an Azure service principal with the DNS Zone Contributor permissions. You can find an example here.

Create the azure.json text file containing the service principal configuration data:

{
  "tenantId": "<EXAMPLE_TENANT_ID>",
  "subscriptionId": "<EXAMPLE_SUBSCRIPTION_ID>",
  "resourceGroup": "<EXAMPLE_RESOURCE_GROUP>",
  "aadClientId": "<EXAMPLE_SP_APP_ID>",
  "aadClientSecret": "<EXAMPLE_SP_PASSWORD>"
}

Create the external-dns-azure-credentials secret in the kof namespace:

kubectl create namespace kof
kubectl create secret generic \
  -n kof external-dns-azure-credentials \
  --from-file azure.json

See external-dns Azure documentation for more details.

Istio#

If you've selected to skip both DNS auto-config now and Manual DNS config later, you can apply these steps to enable the Istio service mesh:

Check the overview in the kof/docs/istio.md and this video:

Create and label the kof namespace to allow Istio to inject its sidecars:

kubectl create namespace kof
kubectl label namespace kof istio-injection=enabled

Install the kof-istio chart to the management cluster:
```
helm upgrade -i --reset-values --wait \
  --create-namespace -n istio-system kof-istio \
  -f global-values.yaml \
  oci://registry.mirantis.com/k0rdent-enterprise/charts/kof-istio --version 1.1.0
```
You may want to customize the collectors by passing values to the kof-istio-child profile for all child clusters at once now, or for each child cluster later. You can also opt for using the default values.

Notice

Currently the kof-istio chart uses ClusterProfiles kof-istio-regional and kof-istio-child.
In the future, however, the KOF team plans to replace them with MultiClusterService objects in the kof-regional and kof-child charts.

If you selected the Istio option, use kof-istio. Otherwise use kof-regional with kof-child. Do not use all 3 charts at the same time.

Management Cluster#

To install KOF on the management cluster, look through the default values of the kof-mothership and kof-operators charts, and apply this example, or use it as a reference:

Install kof-operators as required by kof-mothership:

helm upgrade -i --reset-values --wait \
  --create-namespace -n kof kof-operators \
  -f global-values.yaml \
  oci://registry.mirantis.com/k0rdent-enterprise/charts/kof-operators --version 1.1.0

Create the mothership-values.yaml file:
```
kcm:
  installTemplates: true
```
This enables installation of ServiceTemplates such as cert-manager and kof-storage, making it possible to reference them from the regional and child MultiClusterService objects.
If you want to use a default storage class, but kubectl get sc shows no (default), create it. Otherwise you can use a non-default storage class in the mothership-values.yaml file:
```
global:
  storageClass: <EXAMPLE_STORAGE_CLASS>
```
If kubectl get sc shows nothing or just kubernetes.io/no-provisioner in the PROVISIONER column, apply OpenEBS or similar.

If you've applied the DNS auto-config section, add its information to the kcm: object in the mothership-values.yaml file.

For AWS, add:

  kof:
    clusterProfiles:
      kof-aws-dns-secrets:
        matchLabels:
          k0rdent.mirantis.com/kof-aws-dns-secrets: "true"
        secrets:
          - external-dns-aws-credentials

For Azure, add:

  kof:
    clusterProfiles:
      kof-azure-dns-secrets:
        matchLabels:
          k0rdent.mirantis.com/kof-azure-dns-secrets: "true"
        secrets:
          - external-dns-azure-credentials

This enables Sveltos to auto-distribute the DNS secret to regional clusters.

Two secrets are auto-created by default:
- storage-vmuser-credentials is a secret used by VictoriaMetrics. You don't need to use it directly. It is auto-distributed to other clusters by clusterProfiles here.
- grafana-admin-credentials is a secret that we will use in the Grafana section. It is auto-created here.

Install kof-mothership:

helm upgrade -i --reset-values --wait -n kof kof-mothership \
  -f global-values.yaml \
  -f mothership-values.yaml \
  oci://registry.mirantis.com/k0rdent-enterprise/charts/kof-mothership --version 1.1.0

If you're upgrading from KOF version less than 1.1.0, after upgrade please run:

kubectl apply --server-side --force-conflicts \
-f https://github.com/grafana/grafana-operator/releases/download/v5.18.0/crds.yaml

This is required by grafana-operator release notes.

There is a similar step for each regional cluster on verification step 2.

Wait until the value of VALID changes to true for all ServiceTemplate objects:
```
kubectl get svctmpl -A
```
If you have not applied the Istio section:

Notice

Currently the kof-istio chart uses ClusterProfiles kof-istio-regional and kof-istio-child.
In the future, however, the KOF team plans to replace them with MultiClusterService objects in the kof-regional and kof-child charts.

If you selected the Istio option, use kof-istio. Otherwise use kof-regional with kof-child. Do not use all 3 charts at the same time.
- Look through the MultiClusterService objects in the kof-regional and kof-child charts.
- If your regional clusters already have ingress-nginx and cert-manager services, you can ask the kof-regional chart to not install them by setting Helm values ingress-nginx.enabled and cert-manager.enabled to false.
- You may want to customize collectors for all child clusters at once now, or for each child cluster later, or just use the default values.
- Install these charts into the management cluster with default or custom values:
```
helm upgrade -i --reset-values --wait -n kof kof-regional \
  -f global-values.yaml \
  oci://registry.mirantis.com/k0rdent-enterprise/charts/kof-regional --version 1.1.0

helm upgrade -i --reset-values --wait -n kof kof-child \
  -f global-values.yaml \
  oci://registry.mirantis.com/k0rdent-enterprise/charts/kof-child --version 1.1.0
```
Wait for all pods to show that they're Running:
```
kubectl get pod -n kof
```
Check options to store metrics of the management cluster in the Storing KOF data guide.

Regional Cluster#

To install KOF on the regional cluster, look through the default values of the kof-storage chart, and apply this example for AWS, or use it as a reference:

Set your KOF variables using your own values:

REGION=us-east-2
REGIONAL_CLUSTER_NAME=aws-$REGION
REGIONAL_DOMAIN=$REGIONAL_CLUSTER_NAME.kof.example.com
ADMIN_EMAIL=$(git config user.email)
echo "$REGIONAL_CLUSTER_NAME, $REGIONAL_DOMAIN, $ADMIN_EMAIL"

Use the up-to-date ClusterTemplate, as in:

kubectl get clustertemplate -n kcm-system | grep aws
TEMPLATE=aws-standalone-cp-1-0-12

Compose the regional ClusterDeployment:

For AWS:

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: aws-cluster-identity-cred
  config:
    clusterIdentity:
      name: aws-cluster-identity
      namespace: kcm-system
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      instanceType: t3.large
    workersNumber: 3
    worker:
      instanceType: t3.xlarge
EOF

For Azure:

AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
echo "AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID"

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: azure-cluster-identity-cred
  config:
    clusterIdentity:
      name: azure-cluster-identity
      namespace: kcm-system
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    subscriptionID: $AZURE_SUBSCRIPTION_ID
    location: $REGION
    controlPlaneNumber: 1
    controlPlane:
      vmSize: Standard_A4_v2
    workersNumber: 3
    worker:
      vmSize: Standard_A4_v2
EOF

If you've applied the DNS auto-config section, add it to the .metadata.labels in the regional-cluster.yaml file.

For AWS, add:
```
k0rdent.mirantis.com/kof-aws-dns-secrets: "true"
```
For Azure, add:
```
k0rdent.mirantis.com/kof-azure-dns-secrets: "true"
```
If you've applied the Istio section, update the regional-cluster.yaml file:
- Replace this line:
```
k0rdent.mirantis.com/kof-storage-secrets: "true"
```
  with this line:
```
k0rdent.mirantis.com/istio-role: child
```
  Note that child is not a typo and the kof- prefix is not missing in istio-role. These values make your configuration compatible with possible migration of Istio support from KOF to the upstream k0rdent, where child is a generic term described in the Cluster Deployments concept.
- Delete these lines:
```
k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
```
This ClusterDeployment uses propagation of its .metadata.labels to the resulting Cluster because there are no .spec.config.clusterLabels here. If you add them, please copy .metadata.labels as well.
The aws-standalone-cp template provides the default storage class ebs-csi-default-sc for AWS. The k0rdent quickstart guide provides the default storage class managed-csi for Azure. If you want to use a non-default storage class, add it to the regional-cluster.yaml file in the .spec.config.clusterAnnotations:
```
k0rdent.mirantis.com/kof-storage-class: <EXAMPLE_STORAGE_CLASS>
```

The kof-operator creates and configures PromxyServerGroup and GrafanaDatasource automatically. It uses the endpoints listed below by default. If you want to disable the built-in metrics, logs, and traces to use your own existing instances instead, add custom endpoints to the regional-cluster.yaml file in the .spec.config.clusterAnnotations:

k0rdent.mirantis.com/kof-write-metrics-endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/insert/0/prometheus/api/v1/write
k0rdent.mirantis.com/kof-read-metrics-endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/select/0/prometheus
k0rdent.mirantis.com/kof-write-logs-endpoint: https://vmauth.$REGIONAL_DOMAIN/vls/insert/opentelemetry/v1/logs
k0rdent.mirantis.com/kof-read-logs-endpoint: https://vmauth.$REGIONAL_DOMAIN/vls
k0rdent.mirantis.com/kof-write-traces-endpoint: https://jaeger.$REGIONAL_DOMAIN/collector

If you need a custom http client configuration for PromxyServerGroup and GrafanaDatasource, add it to the regional-cluster.yaml file in the .metadata.annotations. For example:
```
k0rdent.mirantis.com/kof-http-config: '{"dial_timeout": "10s", "tls_config": {"insecure_skip_verify": true}}'
```
The MultiClusterService named kof-regional-cluster configures and installs cert-manager, ingress-nginx, kof-storage, kof-operators, and kof-collectors charts automatically. To pass any custom values to the kof-storage chart or its subcharts, such as victoria-logs-cluster, add them to the regional-cluster.yaml file in the .spec.config.clusterAnnotations. For example:
```
k0rdent.mirantis.com/kof-storage-values: |
  victoria-logs-cluster:
    vlinsert:
      replicaCount: 2
```

Verify and apply the Regional ClusterDeployment:

cat regional-cluster.yaml

kubectl apply -f regional-cluster.yaml

Watch how the cluster is deployed until all values of READY are True:

clusterctl describe cluster -n kcm-system $REGIONAL_CLUSTER_NAME \
  --show-conditions all

Child Cluster#

To install KOF on the actual cluster to be monitored, look through the default values of the kof-operators and kof-collectors charts, and apply this example for AWS, or use it as a reference:

Set your own value below, verifing the variables:

CHILD_CLUSTER_NAME=$REGIONAL_CLUSTER_NAME-child1
echo "$CHILD_CLUSTER_NAME, $REGIONAL_DOMAIN"

Use the up-to-date ClusterTemplate, as in:

kubectl get clustertemplate -n kcm-system | grep aws
TEMPLATE=aws-standalone-cp-1-0-12

Compose the child ClusterDeployment:

For AWS:

cat >child-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: aws-cluster-identity-cred
  config:
    clusterIdentity:
      name: aws-cluster-identity
      namespace: kcm-system
    region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      instanceType: t3.large
    workersNumber: 3
    worker:
      instanceType: t3.medium
EOF

For Azure:

AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
echo "AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID"

cat >child-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: azure-cluster-identity-cred
  config:
    clusterIdentity:
      name: azure-cluster-identity
      namespace: kcm-system
    subscriptionID: $AZURE_SUBSCRIPTION_ID
    location: $REGION
    controlPlaneNumber: 1
    controlPlane:
      vmSize: Standard_A4_v2
    workersNumber: 3
    worker:
      vmSize: Standard_A4_v2
EOF

If you've applied the Istio section, update the child-cluster.yaml file:

replace this line:
```
k0rdent.mirantis.com/kof-storage-secrets: "true"
```
with this line:
```
k0rdent.mirantis.com/istio-role: child
```
This ClusterDeployment uses propagation of its .metadata.labels to the resulting Cluster because there are no .spec.config.clusterLabels here. If you add them, please copy .metadata.labels as well.
The kof-operator discovers the regional cluster by the location of the child cluster. If you have more than one regional cluster in the same AWS region / Azure location / etc., and you want to connect the child cluster to specific regional cluster, add this regional cluster name to the child-cluster.yaml file in the .metadata.labels:
```
k0rdent.mirantis.com/kof-regional-cluster-name: $REGIONAL_CLUSTER_NAME
```
The MultiClusterService named kof-child-cluster configures and installs cert-manager, kof-operators, and kof-collectors charts automatically. To pass any custom values to the kof-collectors chart or its subcharts, such as opencost, add them to the child-cluster.yaml file in the .spec.config. For example:
```
clusterAnnotations:
  k0rdent.mirantis.com/kof-collectors-values: |
    opencost:
      opencost:
        exporter:
          replicas: 2
```
Note: the first opencost key is to reference the subchart, and the second opencost key is part of its values.

Verify and apply the ClusterDeployment:

cat child-cluster.yaml

kubectl apply -f child-cluster.yaml

Watch while the cluster is deployed until all values of READY are True:

clusterctl describe cluster -n kcm-system $CHILD_CLUSTER_NAME \
  --show-conditions all