1.1   ArgoCD¶

Argo CD [1] is a tool for deploying software onto a Kubernetes cluster. Most changes to the Alert Distribution System are done through Argo.

There’s some official SQuARE-maintained documentation for Argo. This section tries to summarize what you’d need to know to work with the Alert Distribution System, but that thorough documentation is worth reading too.

1.2   1Password¶

1Password is a password management tool. LSST IT uses it to distribute passwords, and the SQuARE team has adapted it for managing secrets stored in Kubernetes.

It’s worth reading the documentation in Phalanx on this subject:

• Add a secret with 1Password and VaultSecret
• Updating a secret stored in 1Password and VaultSecret

Managing the Alert Distribution System requires 1Password access. The LSST IT team can grant that access. Then, you’ll also need access to the “RSP-Vault” vault in 1Password, which can be granted by the SQuARE team.

The idea is that credentials are stored in a special 1Password vault with carefully formatted fields. Then you can run the phalanx installer/update_secrets.sh script to copy secrets from 1Password into Vault, which is a tool for encrypting secret data.

In the background, a tool called Vault Secrets Operator copies secret data in Vault and puts it into Kubernetes secrets for use in Kubernetes applications.

This is used to manage the passwords for the Kafka users that can access the alert stream: their passwords are set in 1Password, copied into Vault with the script, and then automatically synchronized into Strimzi KafkaUsers (see also: DMTN-210 3.2.3.1: 1Password, Vault, and Passwords).

1.3   Terraform¶

Terraform is a tool for managing resources stored in Google Cloud through code. The IDF deployment of the alert distribution system uses the github.com/lsst/idf_deploy for its Terraform configuration.

That repository hosts documentation directly in its README. Changes are made entirely through GitHub Actions workflows, so they get applied simply by merging into the main branch.

1.4   Google Cloud Console¶

The Google Cloud project which hosts the IDF deployment of the alert distribution system is “science-platform-int”. You can use the Google Cloud Platform’s console to see some of the things going on inside the system’s cloud resources.

In particular, the Storage Browser can help with identifying anything going wrong with the storage buckets used by the Alert Database, and the Kubernetes Engine UIs might help with exploring the behavior of the deployed systems on Kubernetes.

1.5   Kowl¶

Kowl [2] is a web application that provides a UI for a Kafka broker. It can help with peeking at messages in the Kafka topics, viewing the broker’s configuration, monitoring the state of consumer groups, and more.

Kowl can be run locally using Docker. It requires superuser permissions in the Kafka broker, which can be first retrieved from 1Password (see 2.4   Retrieving Kafka superuser credentials). Then, here’s how to run it locally:

export KAFKA_USER=$(op item get "alert-stream idfint kafka-admin" --fields label=username)
export KAFKA_PASSWORD=$(op item get "alert-stream idfint kafka-admin" --fields label=password)

docker run \
    -p 8080:8080 \
    -e KAFKA_BROKERS=alert-stream-int.lsst.cloud:9094 \
    -e KAFKA_TLS_ENABLED=true \
    -e KAFKA_SASL_ENABLED=true \
    -e KAFKA_SASL_USERNAME=$KAFKA_USER \
    -e KAFKA_SASL_PASSWORD=$KAFKA_PASSWORD \
    -e KAFKA_SASL_MECHANISM=SCRAM-SHA-512 \
    -e KAFKA_SCHEMAREGISTRY_ENABLED=true \
    -e KAFKA_SCHEMAREGISTRY_URLS=https://alert-schemas-int.lsst.cloud \
    quay.io/cloudhut/kowl:master

Once the Kowl container is running, you can view its UI by going to http://localhost:8080.

You should see something like this:

By clicking on a topic, you can see the deserialized messages in the topic. You can expand them by clicking the “+” sign in each row next to the “Value” column. For example:

You can also look at the schema and its versions in the Schema Registry tab:

You can use the Consumer Groups tab to see the position of any consumers. For example, here we can see the Pitt-Google broker:

Kowl has many more capabilities. See the official Kowl documentation [2] for more.

Note: Do note use –network=host, as the current behavior doesn’t allow docker to publish port 8080 and you won’t be able to access the Kowl through the local host.

2.1   Getting kubectl Access¶

While the below instructions are still valid, it is now no longer recommended to use kubectl and instead do everything in the Google Cloud web interface.

1. Install kubectl: https://kubernetes.io/docs/tasks/tools/
2. Install gcloud: https://cloud.google.com/sdk/docs/install
3. Run “gcloud auth login <your google cloud account>”. For example, “gcloud auth login swnelson@lsst.cloud.”
4. Run “gcloud container clusters get-credentials science-platform-int”.

You should now have kubectl access. Try kubectl get kafka --namespace alert-stream-broker to verify. You should see output like this:

-> % kubectl get kafka --namespace alert-stream-broker
NAME           DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS
alert-broker   3                        3                     True

If you haven’t set up your region correctly, you will see this error:

Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=404, message=Not found: projects/science-platform-int-dc5d/zones/us-west1-b/clusters/science-platform-int.

If that happens, open up ~/.config/gcloud/configurations/config_default and set the zone to the suggested zone.

2.2   Secure Password Use¶

In the following sections, you CAN fill in both the username and the password manually on your command line. However, this is not secure and can leave the password/usernames in your command line history. Instead, if you are using 1password, you should use 1passwords command line tool so that you do not directly enter your credentials.

2.3   Running Kowl¶

0. Make sure you have docker installed.

1. Make sure the Docker daemon is running. If using Docker Desktop start up the application.

2. Retrieve Kafka superuser credentials, as described in 2.4   Retrieving Kafka superuser credentials.

3. Run the following:

   export KAFKA_USER=$(op item get "alert-stream idfint kafka-admin" --fields label=username)
   export KAFKA_PASSWORD=$(op item get "alert-stream idfint kafka-admin" --fields label=password)
   
   docker run \
     -p 8080:8080 \
     -e KAFKA_BROKERS=alert-stream-int.lsst.cloud:9094 \
     -e KAFKA_TLS_ENABLED=true \
     -e KAFKA_SASL_ENABLED=true \
     -e KAFKA_SASL_USERNAME=$KAFKA_USER \
     -e KAFKA_SASL_PASSWORD=$KAFKA_PASSWORD \
     -e KAFKA_SASL_MECHANISM=SCRAM-SHA-512 \
     -e KAFKA_SCHEMAREGISTRY_ENABLED=true \
     -e KAFKA_SCHEMAREGISTRY_URLS=https://alert-schemas-int.lsst.cloud \
     quay.io/cloudhut/kowl:master
   

3. Go to http://localhost:8080

2.4   Retrieving Kafka superuser credentials¶

The superuser has access to do anything. Be careful with these credentials! To find the credentials:

1. Log in to 1Password in the LSST IT account.
2. Go to the “RSP-Vault” vault.
3. Search for “alert-stream idfint kafka-admin”.

2.5   Retrieving development credentials¶

This user only has limited permissions, mimicking those of a community broker.

1. Log in to 1Password in the LSST IT account.
2. Go to the “RSP-Vault” vault.
3. Search for “alert-stream idfint rubin-communitybroker-idfint”.

3.1   Testing connectivity¶

First, get the set of developer credentials (2.5   Retrieving development credentials).

Then, use one of the example consumer applications listed in sample_alert_info/examples. These will show whether you’re able to connect to the Kafka stream and receive sample alert packets, as well as whether you’re able to retrieve schemas from the Schema Registry.

3.2   Checking disk usage¶

First, check how much disk is used by Kafka:

1. Run Kowl, following the instructions in 2.2   Secure Password Use.

2. Navigate to the brokers view at http://localhost:8080/brokers.

   You should see the amount of disk used by each broker in the right-most column under “size.”

Next, check how much is requested in the persistent volume claims used by the Kafka brokers:

3. Ensure you have kubectl access (2.1   Getting kubectl Access).

4. Run kubectl get pvc --namespace alert-stream-broker. You should see output like this:

   -> % kubectl get pvc -n alert-stream-broker
   NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   data-0-alert-broker-kafka-0     Bound    pvc-e5bf9fb1-e763-4c03-8294-b81a6955bde3   1500Gi     RWO            standard       77d
   data-0-alert-broker-kafka-1     Bound    pvc-c289fc0d-39a0-44b1-b073-2aab5c47ba3a   1500Gi     RWO            standard       77d
   data-0-alert-broker-kafka-2     Bound    pvc-6307f422-0448-45bd-985b-f7e559e54bb9   1500Gi     RWO            standard       77d
   data-alert-broker-zookeeper-0   Bound    pvc-bd8bb38f-a5d3-47f9-a9a1-13c66f04f80e   1000Gi     RWO            standard       77d
   data-alert-broker-zookeeper-1   Bound    pvc-01463914-9b1f-49bd-992f-de0b6b0284ca   1000Gi     RWO            standard       77d
   data-alert-broker-zookeeper-2   Bound    pvc-eb37bbaa-cc49-4541-baf4-6f2444330d6f   1000Gi     RWO            standard       77d
   

3.3   Checking consumer group status¶

1. Run Kowl, following the instructions in 2.2   Secure Password Use.
2. Navigate to the consumer group view at http://localhost:8080/groups

There should be an entry for each consumer group that is connected or has connected recently.

The “Coordinator” column indicates which of the three Kafka broker nodes is used for coordinating the group’s partition ownership.

The “Members” column indicates the number of currently-active processes which are consuming data.

The “Lag” column indicates how many messages are unread by the consumer group.

3.4   Checking logs¶

In general, logs are available on the Google Cloud Log Explorer UI.

To access them:

1. Log in to the Google Cloud console at https://console.cloud.google.com.

2. Navigate to the Log Explorer UI, https://console.cloud.google.com/logs/query

3. Enter a search query. For example:

   This will bring up all logs from Kafka brokers:

   

There are additional “Log fields” on the left column. You can use these to filter to a single one of the three brokers via the “Pod name” field.

You can pick a different time range by clicking on “Last 1 hour” in the top right:

See also: the GCP Log Explorer documentation: https://cloud.google.com/logging/docs/view/logs-viewer-interface

Each of the subsections lists search queries that can be used to filter logs.

resource.type="k8s_container"
resource.labels.container_name="kafka"
resource.labels.namespace_name="alert-stream-broker"

resource.type="k8s_container"
resource.labels.namespace_name="strimzi"

resource.type="k8s_container"
resource.labels.namespace_name="strimzi-registry-operator"

resource.type="k8s_container"
resource.labels.pod_name:"alert-schema-registry"
resource.labels.namespace_name="alert-stream-broker"

resource.type="k8s_container"
resource.labels.pod_name:"alert-database"
resource.labels.namespace_name="alert-stream-broker"

resource.type="k8s_container"
resource.labels.pod_name="alert-stream-simulator"
resource.labels.namespace_name="alert-stream-broker"

4.1   Sharing passwords¶

1. Log in to 1Password in the LSST IT account.

2. Go to the “RSP-Vault” vault.

3. Search for the username of the account you want to share.

4. Click on the 3-dot menu in the top right and choose “Share…”:

   

   This will open a new browser window for a sharing link.

5. Set the duration and availability as desired, and click “Get Link to Share”:

   

Share the link as you see fit.

Shared links can also be revoked; see 1Password Documentation for more.

4.2   Changing passwords¶

1. Log in to 1Password in the LSST IT account.
2. Go to the “RSP-Vault” vault.
3. Search for the username of the account you want to modify.
4. Click on the password field. Generate a new password and set it, and save your changes.
5. Follow the instructions in Phalanx: Updating a secret stored in 1Password and VaultSecret.

Then verify that the change was successful by checking it in Argo.

1. Log in to Argo (see also 1.1.1   Accessing Argo).
2. Navigate to the “alert-stream-broker” application.
3. In the “filters” on the left side, search for your targeted username in the “Name” field. You should see a filtered set of resources now.
4. Click on the “secret” resource and check that it has an “updated” timestamp that is after you made your changes. If not, delete the “Secret” resource; it will be automatically recreated quickly. Once recreated, the user’s password will be updated automatically.

If this seems to be having trouble, consider checking:

• the Vault Secrets Operator logs to make sure it is updating secrets correctly
• the Strimzi Entity Operator logs to make sure they are updating user accounts correctly
• the Kafka broker logs to make sure it’s healthy

4.3   Adding a new user account¶

First, generate new credentials for the user:

1. Log in to 1Password in the LSST IT account.

2. Go to the “RSP-Vault” vault.

3. Create a new secret.

   1. Name it “alert-stream idfint <username>”.
   2. Set the “Username” field to <username>.
   3. Set the “Password” field to something autogenerated.
   4. Add a field named “generate_secrets_key”. Set its value to “alert-stream-broker <username>-password”
   5. Add a field named “environment”. Set its value to “data-int.lsst.cloud”

   If you’re running in a different environment than the IDF integration environment, replaced “idfint” and “data-int.lsst.cloud” with appropriate values.

4. Sync the secret into Vault following the instructions in Phalanx documentation.

Second, add the user to the configuration for the cluster:

1. Make a change to github.com/lsst-sqre/phalanx’s services/alert-stream-broker/values-idfint.yaml file. Add the new user to the list of users under alert-stream-broker.users: https://github.com/lsst-sqre/phalanx/blob/bb417e80e0d9d1148da6edccae400eec006576e1/services/alert-stream-broker/values-idfint.yaml#L33-L73

   Make sure you use the same username, and grant it read-only access to the alerts-simulated topic by setting readonlyTopics: ["alerts-simulated"] just like the other entries.

   If more topics should be available, add them. If running in a different environment than the IDF integration environment, modify the appropriate config file, not values-idfint.yaml.

2. Make a pull request with your changes, and make sure it passes automated checks, and get it reviewed.

3. Merge your PR. Wait a few minutes (perhaps 10) for Argo to pick up the change.

4. Log in to Argo CD.

5. Navigate to the ‘alert-stream-broker’ application.

6. Click “sync” and leave all the defaults to sync your changes, creating the new user.

Verify that the new KafkaUser was created by using the filters on the left side to search for the new username.

Verify that the user was added to Kafka by using Kowl and going to the “Access Control List” section (see 2.2   Secure Password Use).

Optionally verify that access works using a method similar to that in 3.1   Testing connectivity.

4.4   Removing a user account¶

1. Delete the user from the list in github.com/lsst-sqre/phalanx’s services/alert-stream-broker/values-idfint.yaml file.
2. Make a pull request with this change, and make sure it passes automated checks, and get it reviewed.
3. Merge your PR.
4. Delete the user’s credentials from 1Password in the RSP-Vault vault of the LSST IT account. You can find the credentials by searching by username.
5. Log in to Argo CD.
6. Navigate to the ‘alert-stream-broker’ application.
7. Click “sync”. Click the “prune” checkbox to prune out the defunct user. Apply the sync.

Verify that the user was removed from Kafka by using Kowl and going to the “Access Control List” section (see 2.2   Secure Password Use). The user shouldn’t be in the ACLs anymore.

4.5   Granting users read-only access to a new topic¶

1. Make a change to github.com/lsst-sqre/phalanx’s services/alert-stream-broker/values-idfint.yaml file. In the list of users under alert-stream-broker.users, add the new topic to the readonlyTopics list for each user that should have access.
2. Make a pull request with your changes, and make sure it passes automated checks, and get it reviewed.
3. Merge your PR. Wait a few minutes (perhaps 10) for Argo to pick up the change.
4. Log in to Argo CD.
5. Navigate to the ‘alert-stream-broker’ application.
6. Click “sync” and leave all the defaults to sync your changes, modifying access.

Verify that the change worked by using Kowl and going to the “Access Control List” section (see 2.2   Secure Password Use). There should be matching permissions with Resource=TOPIC, Permission=ALLOW, and Principal being the users who were granted access.

4.6   Adding a new Kafka topic¶

1. Add a new KafkaTopic resource to the templates directory in one of the charts that composes the alert-stream-broker service. This will be in the alert-stream-broker/charts repository. For example, there is a KafkaTopic resource in the alert-stream-simulator/templates/kafka-topics.yaml file.

   These files use the Helm templating language. See The Chart Template Developer’s Guide for more information on this language.

   Strimzi’s documentation (“5.2.1: Kafka topic resource”) may be helpful in configuring the topic. The schema for KafkaTopic resources has a complete reference at 11.2.90: KafkaTopic schema reference.

   Pick the chart that is most relevant to the topic you are adding. If it is not relevant to any particular chart, use the general charts/alert-stream-broker chart.

2. Increment the version of the chart by updating the version field of its Chart.yaml file. For example, this line of the alert-stream-simulator chart.

3. Make a pull request with your changes to alert-stream-broker/charts, and make sure it passes automated checks, and get it reviewed. Merge your PR.

4. Next, you’ll update the services/alert-stream-broker/Chart.yaml file to reference the new version number of the chart you have updated. For example, this line would need to be updated if you were adding a topic to the alert-stream-simulator.

5. Make a pull request with your changes to github.com/lsst-sqre/phalanx, and make sure it passes automated checks, and get it reviewd. Merge your PR.

6. Wait a few minutes (perhaps 10) for Argo to pick up the change to Phalanx.

7. Log in to Argo CD.

8. Navigate to the ‘alert-stream-broker’ application.

9. Click ‘sync’ and leave all the defaults to sync your changes, creating the new topic.

Verify that the change worked by using Kowl and going to the “Topics” section (see 2.2   Secure Password Use). There should be a new topic created.

To let users read from the topic, see 4.5   Granting users read-only access to a new topic.

4.7   Granting Alert DB access¶

Alert DB access is governed by membership in GitHub organizations and teams.

The list of permitted GitHub groups for the IDF integration environment is in the services/gafaelfawr/values-idfint.yaml file in github.com/lsst-sqre/phalanx.

As of this writing, that list is composed of ‘lsst-sqre-square’ and ‘lsst-sqre-friends’, so any users who wish to have access need to be added to the “square” or “friends” teams in the lsst-sqre GitHub organization.

Invite a user to join one of those groups to grant access.

To change the set of permitted groups, modify the services/gafaelfawr/values-idfint.yaml file to change the list under the read:alertdb scope. Then, sync the change to Gafaelfawr via Argo CD.

5.1   Deploying a change with Argo¶

In general, to make any change with ArgoCD, you update Helm charts, update Phalanx, and then “sync” the alert-stream-application:

1. Make desired changes to Helm charts, if required, in alert-stream-broker/charts. Note that any changes to Helm charts always require the version to be updated.
2. Merge your Helm chart changes.
3. Update the services/alert-stream-broker/Chart.yaml file to reference the new version number of the chart you have updated, if you made any Helm chart changes.
4. Update the services/alert-stream-broker/values-idfint.yaml file to pass in any new template parameters, or make modifications to existing ones.
5. Merge your Phalanx changes.
6. Wait a few minutes (perhaps 10) for Argo to pick up the change to Phalanx.
7. Log in to Argo CD at https://data-int.lsst.cloud/argo-cd.
8. Navigate to the ‘alert-stream-broker’ application.
9. Click ‘sync’ to synchronize your changes.

5.2   Updating the Kafka version¶

The Kafka version is set in the alert-stream-broker/templates/kafka.yaml file in services/alert-stream-broker. It is parameterized through the kafka.version value in the alert-stream-broker chart, which defaults to “2.8”.

When upgrading the Kafka version, you also may need to update the kafka.logMesageFormatVersion and kafka.interBrokerProtocolVersion. These change slowly, but old values can be incompatible with new Kafka versions. See Strimzi documentation on Kafka Versions to be sure.

So, to update the version of Kafka used, update the services/alert-stream-broker/values-idfint.yaml file in github.com/lsst-sqre/phalanx. Under alert-stream-broker, then under kafka, add a value: version: <whatever you want>. If necessary, also set logMessageFormatVersion and interBrokerProtocolVersion here.

Then, follow the steps in 5.1   Deploying a change with Argo to apply these changes.

See also: the Strimzi Documentation’s “9.5: Upgading Kafka”.

5.3   Updating the Strimzi version¶

First, you probably want to read the Strimzi Documentation’s “9. Upgrading Strimzi”.

The Strimzi version is governed by the version referenced in github.com/lsst-sqre/phalanx’s services/strimzi/Chart.yaml file. Update that version, and do anything else recommended by Strimzi in their documentation, such as changes to resources.

Then, apply the change in a way similar to that described in 5.1   Deploying a change with Argo. Note though that you’ll be synchronizing the ‘strimzi’ application in Argo, not the ‘alert-stream-broker’ application in Argo.

5.4   Resizing Kafka broker disk storage¶

Some reference reading:

• DMTN-210’s section 3.2.1.3: Storage.
• “Persistent storage improvements”

Change the alert-stream-broker.kafka.storage.size value in services/alert-stream-broker/values-idfint.yaml in github.com/lsst-sqre/phalanx. This is the amount of disk space per broker instance.

Apply the change, as described in 5.1   Deploying a change with Argo.

This may take a little while to apply, since it is handled through the asynchronous Kafka operator, which reconciles storage size every few minutes. When it starts reconciling, it rolls the change out gradually across the Kafka cluster to maintain availability.

Note that storage sizes can only be increased, never decreased.

5.5   Updating the alert schema¶

For background, you might want to read DMTN-210’s section 3.4.4: Schema Synchronization Job.

The high-level steps are to:

• Commit your changes in the lsst/alert_packet repository, obeying its particular versioning system
• Build a new lsstdm/lsst_alert_packet container
• Publish a new lsst-alert-packet Python package
• Load the schema into the schema registry, incrementing the Schema ID
• Update the alert-stream-simulator to use the new Python package and new schema ID

-> % docker run --rm lsstdm/lsst_alert_packet:w.2022.04 'syncLatestSchemaToRegistry.py --help'
usage: syncLatestSchemaToRegistry.py [-h]
                                     [--schema-registry-url SCHEMA_REGISTRY_URL]
                                     [--subject SUBJECT]

optional arguments:
  -h, --help            show this help message and exit
  --schema-registry-url SCHEMA_REGISTRY_URL
                        URL of a Schema Registry service
  --subject SUBJECT     Schema Registry subject name to use

-> % docker run --rm lsstdm/alert-stream-simulator:v1.2.1 'rubin-alert-sim -h'
usage: rubin-alert-sim [-h] [-v] [-d]
                       {create-stream,play-stream,print-stream} ...

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         enable info-level logging (default: False)
  -d, --debug           enable debug-level logging (default: False)

subcommands:
  {create-stream,play-stream,print-stream}
    create-stream       create a stream dataset to be run through the
                        simulation.
    play-stream         play back a stream that has already been created
    print-stream        print the size of messages in the stream in real time

5.6   Changing the sample alert data¶

The sample alert data used by the alert stream simulator is set in a Makefile:

.PHONY: datasets
datasets: data/rubin_single_ccd_sample.avro data/rubin_single_visit_sample.avro

data:
        mkdir -p data

data/rubin_single_ccd_sample.avro: data
        wget --no-verbose --output-document data/rubin_single_ccd_sample.avro https://lsst.ncsa.illinois.edu/~ebellm/sample_precursor_alerts/latest_single_ccd_sample.avro

data/rubin_single_visit_sample.avro: data
        wget --no-verbose --output-document data/rubin_single_visit_sample.avro https://lsst.ncsa.illinois.edu/~ebellm/sample_precursor_alerts/latest_single_visit_sample.avro

The last two show what’s happening. The sample alerts are downloaded from https://lsst.ncsa.illinois.edu/~ebellm/sample_precursor_alerts/latest_single_visit_sample.avro.

The sample alerts could be retrieved from anywhere else. The important things are that they should be encoded in Avro Object Container File format (that is, with all alerts in one file, preceded by a single instance of the Avro schema), and that they should represent a single visit of alert packet data.

Make changes to the makefile to get data from somewhere else, and then merge your changes. Make a git tag using the format vX.Y.Z, for example v1.3.10, and push that git tag up. This will trigger a build job for the container using the new tag.

Next, copy that tag into charts/alert-stream-simulator/values.yaml, and follow the instructions from 5.1   Deploying a change with Argo. This will configure the alert stream simulator to use the new alert data, publishing it every 37 seconds.

5.7   Deploying on a new Kubernetes cluster on Google Kubernetes Engine¶

Deploying on a new Kubernetes cluster will take a lot of steps, and has not been done before, so this section is somewhat speculative.

  {
    name = "kafka-pool"
    machine_type = "n2-standard-32"
    node_locations     = "us-central1-b"
    local_ssd_count    = 0
    auto_repair        = true
    auto_upgrade       = true
    preemptible        = false
    image_type         = "cos_containerd"
    enable_secure_boot = true
    disk_size_gb       = "500"
    disk_type          = "pd-standard"
    autoscaling        = true
    initial_node_count = 1
    min_count          = 1
    max_count          = 10
  }
]

node_pools_labels = {
  core-pool = {
    infrastructure = "ok",
    jupyterlab = "ok"
  },
  dask-pool = {
    dask = "ok"
  },
  kafka-pool = {
    kafka = "ok"
  }
}

node_pools_taints = {
  core-pool = [],
  dask-pool = []
  kafka-pool = [
    {
      effect = "NO_SCHEDULE"
      key = "kafka",
      value = "ok"
    }
  ]
}

# Project
environment = "int"
project_id  = "science-platform-int-dc5d"

# In integration, only keep 4 weeks of simulated alert data.
purge_old_alerts  = true
maximum_alert_age = 28

writer_k8s_namespace           = "alert-stream-broker"
writer_k8s_serviceaccount_name = "alert-database-writer"
reader_k8s_namespace           = "alert-stream-broker"
reader_k8s_serviceaccount_name = "alert-database-reader"

# Increase this number to force Terraform to update the int environment.
# Serial: 2

-> % kubectl get service  -n alert-stream-broker
NAME                                    TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                               AGE
alert-broker-kafka-0                    LoadBalancer   10.130.20.152   35.239.64.164    9094:31402/TCP                        78d
alert-broker-kafka-1                    LoadBalancer   10.130.23.65    34.122.165.155   9094:31828/TCP                        78d
alert-broker-kafka-2                    LoadBalancer   10.130.21.82    35.238.120.127   9094:31070/TCP                        78d
alert-broker-kafka-bootstrap            ClusterIP      10.130.20.156   <none>           9091/TCP,9092/TCP,9093/TCP            78d
alert-broker-kafka-brokers              ClusterIP      None            <none>           9090/TCP,9091/TCP,9092/TCP,9093/TCP   78d
alert-broker-kafka-external-bootstrap   LoadBalancer   10.130.16.127   35.188.169.31    9094:30118/TCP                        78d
alert-broker-zookeeper-client           ClusterIP      10.130.25.236   <none>           2181/TCP                              78d
alert-broker-zookeeper-nodes            ClusterIP      None            <none>           2181/TCP,2888/TCP,3888/TCP            78d
alert-schema-registry                   ClusterIP      10.130.27.137   <none>           8081/TCP                              76d
alert-stream-broker-alert-database      ClusterIP      10.130.27.41    <none>           3000/TCP                              21d

5.8   Deploying on a new Kubernetes cluster off of Google¶

Deploying to a new Kubernetes cluster off of Google will require all the same steps as described in the previous section, but with a few additional wrinkles.

First, the alert-stream-broker chart uses the “load balancer” service type to provide external internet access to the Kafka nodes. Load balancer services are very platform-specific; on Google it corresponds to creation of TCP Load Balancers. On a non-Google platform, it might work very differently.

One option would be to use the targeted platform’s load balancers. Another option is to use Node Ports or Ingresses instead. The 5-part Strimzi blog post series “Accessing Kafka” goes into detail about these options.

Second, the alert database uses Google Cloud Storage buckets to store raw alert and schema data. This would need to be replaced with something appropriate for the targeted environment. The requirements are made clear in the storage.py files of the github.com/lsst-dm/alert_database_ingester and github.com/lsst-dm/alert_database_server repositories. An implementation would need to fulfill the abstract interface provided in that file.

There may be more requirements, but these are certainly to need investigation if you’re planning to move to a different Kubernetes provider.

5.9   Changing the schema registry hostname¶

The Schema Registry’s hostname is controlled by the ‘hostname’ value passed in to charts/alert-stream-schema-registry. Updating that will update the hostname expected by the service.

In addition, a new DNS record will need to be created by whoever is provisioning DNS for the target environment. For the INT IDF environment, that’s SQuARE. It should route the new hostname to the ingress IP address.

Finally, the new schema registry needs to be passed in to the alert database in its ingester.schemaRegistryURL value.

See also: 5.7.3   Provision the DNS for the schema registry.

5.10   Changing the Kafka broker hostnames¶

Kafka broker hostnames can be changed by modifying the values passed in to charts/alert-stream-broker. Once changed, the broker will not work until DNS records are also updated.

See also: 5.7.6   Provisioning DNS records.

5.11   Changing the alert database URL¶

The alert database’s URL is based off of that of the cluster’s main Gafaelfawr ingress, so it cannot be changed entirely. However, it uses a path prefix, which can be changed. This path prefix is controlled by a value passed in to the alert database chart.

5.12   Changing the Kafka hardware¶

To change the hardware used by Kafka, change the nodes used in the node pool. This is set in the terraform configuration in environment/deployments/science-platform/env/integration-gke.tfvars:

{
  name = "kafka-pool"
  machine_type = "n2-standard-32"
  node_locations     = "us-central1-b"
  local_ssd_count    = 0
  auto_repair        = true
  auto_upgrade       = true
  preemptible        = false
  image_type         = "cos_containerd"
  enable_secure_boot = true
  disk_size_gb       = "500"
  disk_type          = "pd-standard"
  autoscaling        = true
  initial_node_count = 1
  min_count          = 1
  max_count          = 10
}

Change this, and apply the terraform change.

This may cause some downtime as the kafka nodes are terminated and replaced with new ones, evicting the Kafka brokers, but this isn’t known for certain.

References