Putting it all together

For the sake of this brief  I have picked two different public domains registrars, namely Google Domains and Gandi.net and two different external DNS providers, namely: Google Cloud DNS and Azure DNS.

When you provision a kyma cluster it gets assigned a secure wildcard public domain name, for instance: .kyma.ondemand.com, which is entirely managed by SAP.

The built-in kyma-system/kyma-gateway is configured with this default domain.

Likewise, your kyma cluster’s API server, namely: https://api.<shoot-name>.kyma.ondemand.com, is bound to that domain as well. Thus, a cluster domain cannot be changed once a cluster has been provisioned.

Q. But what if you wanted to have your own domain with a kyma cluster gateway?  In other words, how to set up a domain for workloads across a kyma cluster?

A. Here you go. You will need to accomplish these three tasks, namely:

I shall endeavour to walk you step by step throughout each of these three tasks. However, if you are network-fluent and have complied with the Task A and Task B requirements, you may jump directly to Task C.

Task A. Bring Your Own Domain (BYOD).

1a. Custom domains with Google Domains

Google Domains is a domain registrar. For instance, you may look up your registered domain names at: https://domains.google.com/registrar/

However, as Google Domains is not a DNS (domain name service) provider per se. Thus you will need to bring the DNS servers names from your established external DNS provider.

Steps	Google Domains registered domain’s custom DNS settings
Goto DNS settings and switch to Custom name servers. Please make sure the domain, you are about to modify the DNS settings of, is not used for your websites or email. However, If that were the case, you might want to register a different domain dedicated to your kyma workloads
The custom name servers will come from either the Google Cloud DNS or Azure DNS provider. As Google Domains does not support APIs you will have apply them manually.
After saving the new DNS settings these will be effective after a short while.

1b. Custom domains with Gandi.net

Gandi.net is a popular domain registrar and a DNS provider. However, as Gandi is not a supported external DNS provider I had to bring the DNS name servers from an AzureDNS zone manually as well as depicted below:

AzureDNS zone:

Gandi.net domains registrar

Task B. Set up an external DNS provider. Create a DNS zone.

In order to be able to use one of your registered domains with SAP BTP, Kyma runtime, the first thing you need to do with either Google CloudDNS or AzureDNS is to create a DNS zone for your domain and then copy the DNS servers names to your domains registrar.

1a. Create Google Cloud DNS zone within your GCP project

In order to be able to use one your Google Domains registered domains with SKR you will need to set up a DNS zone to host the DNS records for a chosen domain name with Google Cloud DNS as follows:

Quickstart: Set up DNS records for a domain name with Cloud DNS | Google Cloud Docs

Steps	Google Network Service/Cloud DNS
Goto Network Service/Cloud DNS within your project.
Create a DNS zone for keeping your Google Domains domain’s DNS records.
The default SOA and NS records got created.However, additional DNS records for a chosen domain name will be created when configuring the SKR cluster DNS resources. Thus no need to create the records manually.
Copy the following name servers to your registrar domain’s custom DNS settings (via a click on the Registrar setup button or via Equivalent REST shown below).

{
  "cloudLoggingConfig": {
    "enableLogging": false
  },
  "creationTime": "<creationTime>",
  "description": "quovadis-btp",
  "dnsName": "quovadis-btp.com.",
  "fingerprint": "<fingerprint>",
  "id": "<id>",
  "location": "global",
  "name": "quovadis-btp",
  "nameServers": [
    "ns-cloud-**.googledomains.com.",
    "ns-cloud-**.googledomains.com.",
    "ns-cloud-**.googledomains.com.",
    "ns-cloud-**.googledomains.com."
  ],
  "visibility": "PUBLIC"
}

1b. Create Azure DNS zone within your Azure subscription resource group.

In order to be able  create a DNS zone you need to have a valid Azure subscription and have created a resource group as depicted below:

Azure DNS zone creation steps:

{
    "id": "/subscriptions/<subscriptionId>/resourceGroups/quovadis/providers/Microsoft.Network/dnszones/quovadis-anywhere.com",
    "name": "quovadis-anywhere.com",
    "type": "Microsoft.Network/dnszones",
    "etag": "<etag>",
    "location": "global",
    "tags": {},
    "properties": {
        "maxNumberOfRecordSets": 10000,
        "maxNumberOfRecordsPerRecordSet": null,
        "nameServers": [
            "ns1-**.azure-dns.com.",
            "ns2-**.azure-dns.net.",
            "ns3-**.azure-dns.org.",
            "ns4-**.azure-dns.info."
        ],
        "numberOfRecordSets": 2
    }
}

2. Create an API access to a DNS zone

The kyma cluster will need an application level (API) access to a DNS zone. That’s a pre-requisite for enabling the DNS provisioning automation on kyma side.

• With Google CloudDNS you will need to create a service account with a service key.
• With AzureDNS you will have to create an Azure AD application and service principal that can access AzureDNS zone

2a. Create a GooglePlatform Service Account and a service key

A service key is required for the kyma cluster to get access to the DNS records of the CloudDNS managed domain. The service key must allow for DNS records modifications.

Steps	Google IAM and admin/Service Accounts
In your Google Cloud project goto Google IAM and admin/Service Accounts and create a service account
The service account has been created
Create a service key with the service account credentials. The service key will need to passed to the kyma cluster namespace as a secret

2b. create an Azure AD application and service principal

The main tutorial that explains how to do it (especially if you do not have access to a Windows machine) is here:

• https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

For instance, here go AzureDNS API credentials, namely azure.json:

{
  "tenantId": "<tenantId>",
  "subscriptionId": "<subscriptionId>",
  "resourceGroup": "quovadis",
  "aadClientId": "8e4ed817-c1b0-4cab-89a0-************",
  "aadClientSecret": "c91z*************************"
}

Task C. Configure kyma cluster DNS resources

As usual the best is to refer to the DNS providers source documentation from the Gardener project.

The External-DNS-Management project provides examples with more details for DNSProviders (30-provider-<provider-name>.yaml) and credential Secrets (20-secret-<provider-name>.yaml)  for all supported provider types.

All the DNS resources are namespaced. Thus, it’s probably a good idea to have a dedicated namespace per DNS provider in a kyma cluster.

0. Create a DNS automation namespace

Let’s call this namespace: conn-proxy. Goto namespace configuration to set up the following DNS resources.

1. Create an external DNS provider secret

The DNS provider secret is required for DNS resources automation.

There are two ways to create secrets. Either using the kyma dashboard secret presets or with kubectl as shown below:

$ kubectl create secret generic quovadis-btp-dns --namespace=conn-proxy--from-file=serviceaccount.json=<project name>-11111111111.json
secret/quovadis-btp-dns created

2. Create a DNS provider and a DNS entry

apiVersion: dns.gardener.cloud/v1alpha1 kind: DNSProvider metadata: name: quovadis-bpt-dns-provider namespace: conn-proxy labels: app.kubernetes.io/name: quovadis-bpt-dns-provider annotations: dns.gardener.cloud/class: garden spec: domains: include: - quovadis-btp.com - '*.quovadis-btp.com' exclude: [] secretRef: name: quovadis-btp-dns namespace: conn-proxy type: google-clouddns

apiVersion: dns.gardener.cloud/v1alpha1 kind: DNSEntry metadata: name: quovadis-bpt-dns-entry namespace: conn-proxy labels: app.kubernetes.io/name: quovadis-bpt-dns-entry annotations: dns.gardener.cloud/class: garden spec: dnsName: '*.quovadis-btp.com' targets: - **.**.**.** text: [] ttl: 600

3. Create an Issuer with ACME (Automated Certificate Management Environment)

The issuer resource is required to sign the certificate for the domain.

apiVersion: cert.gardener.cloud/v1alpha1
kind: Issuer
metadata:
  name: quovadis-btp-ca-issuer
  namespace: conn-proxy
  labels:
    app.kubernetes.io/name: quovadis-btp-ca-issuer
spec:
  requestsPerDayQuota: 0
  acme:
    server: 'https://acme-v02.api.letsencrypt.org/directory'
    email: <email address>
    autoRegistration: true
    privateKeySecretRef:
      name: ''
      namespace: conn-proxy
    domains:
      include:
        - quovadis-btp.com
        - '*.quovadis-btp.com'

4. Create a certificate resource

apiVersion: cert.gardener.cloud/v1alpha1 kind: Certificate metadata: labels: app.kubernetes.io/name: quovadis-btp-cert name: quovadis-btp-cert namespace: istio-system spec: commonName: '*.quovadis-btp.com' issuerRef: name: quovadis-btp-ca-issuer namespace: conn-proxy secretRef: name: tls-quovadis-btp-cert namespace: istio-system issuerRef: cluster: target name: quovadis-btp-ca-issuer namespace: conn-proxy

apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: labels: app.kubernetes.io/name: quovadis-gateway name: quovadis-gateway namespace: conn-proxy spec: selector: istio: ingressgateway servers: - hosts: - '*.quovadis-btp.com' port: name: quovadis-https number: 443 protocol: HTTPS tls: credentialName: tls-quovadis-btp-cert mode: SIMPLE - hosts: - '*.quovadis-btp.com' port: name: quovadis-http number: 80 protocol: HTTP tls: httpsRedirect: true

apiVersion: gateway.kyma-project.io/v1alpha1
kind: APIRule
metadata:
  labels:
    app.kubernetes.io/name: quovadis-s4
  name: quovadis-s4
  namespace: conn-proxy
spec:
  gateway: quovadis-gateway.conn-proxy.svc.cluster.local
  rules:
    - accessStrategies:
        - config:
            jwks_urls:
              - >-
                https://<zone>.authentication.<region>.hana.ondemand.com/token_keys
            trusted_issuers:
              - >-
                https://<zone>.authentication.<region>.hana.ondemand.com/oauth/token
          handler: jwt
      methods:
        - PUT
        - POST
        - PATCH
        - HEAD
        - GET
        - DELETE
      path: /.*
  service:
    host: ***.quovadis-btp.com
    name: test-s4
    port: 80

and voila:

Good to know:

• The gateway, namely: quovadis-gateway.conn-proxy.svc.cluster.local, can be used from any namespace across your kyma cluster

Conclusion

I hope you have enjoyed reading this blog post.

You have seen how to expose your Kyma cluster workloads with istio gateways over a custom domain.

Whether this has been easy or not I shall leave it to your judgement. The good news is that this is a one-time effort and that a one single wildcard custom domain can be used for any workload across the SKR cluster.

Last but not least, to stay up to date with the latest news on SAP BTP, Kyma runtime make sure to bookmark our Kyma topic page.

Appendix

ACME Directory URL: https://acme-v02.api.letsencrypt.org/directory

{
  "ddLcOs-N9Uo": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

Additional resources

1. A hands-on tutorial leveraging Cloud-fare DNS | SAP Samples

2. New with SAP BTP, Kyma Runtime: Seamlessly connect your website or microservice to a custom domain using Istio Service Mesh | SAP Blogs

3. a video tutorial from SAP HANA Academy

4. Beyond the Gardener project there are other public resources that describe the external DNS providers configurations and settings.

5. https://practicaldatascience.co.uk/data-engineering/how-to-create-a-google-service-account-client-secrets-json-key

6. https://stackoverflow.com/questions/70774865/add-a-service-account-json-to-kubectl-secrets

7. https://stackexchange.github.io/dnscontrol/providers/azuredns