Conclusion Xforce Tech Playground –

Kubecon 2026 relived

Mon, 22 Jun 2026 00:00:00 +0000

Introduction

I like big conferences: you can get a lot of knowledge in a very short time. During the Kubecon conference in Amsterdam in March 2026, I maintained a list of what I wanted to learn after the conference. In the weeks after Kubecon, I tried to recreate the examples in the AWS play environment. In this case: a vanilla implementation of Kubernetes with one control node and two worker nodes. In this blog some of the lessons I learned.

You can deploy the examples in your own AWS environment as well, this is the link to the repository Go to the aws-deployment directory, copy the setenv.template.sh file to setenv.sh and change the values in the file. Then start the environment with

bash login.sh
bash start-k8s.sh

You can delete the environment by

bash stop-k8s.sh

01 - Gateway API instead of Ingress

It was an official moment in the Kubecon presentation: the archival of the ingress-nginx repository. We now should migrate to the new Gateway API implementations. There are two tools that can help you: the first one is called ingress2gateway and it can help you in migrating the current ingress-nginx implementations to a Gateway API implementations. The second one is an implementation wizard: when you don’t know which implementation support which features you can click on the features that you must have, or which ones are nice to haves, and then a list of tools that support these features comes up.

In the talk “Gateway API: Bridging the Gap from Ingress to the Future” ( slides, video) the presenters showed both the way of working with new ListenerSets, TLSRoutes in both Terminate and Passthrough mode, mTLS, HTTP2 Connection coalescing and the Gateway client certificate selection API.

In my example repository I have examples for the new ListenerSet and the two ways of working with TLSRoutes.

02 - Refresh secrets

The ArgoCD team gets a lots of questions to implement support for secrets in the ArgoCD environment. They don’t want to implement it, because secrets should be kept in a vault (f.e. Hashicorp Vault, Azure KeyVault, AWS Secrets Manager, etc). When the password changes, the External Secrets Operator will keep the secret in Kubernetes up-to-date automatically. As an developer, you can use libraries like fsnotify (for go) or watchdog (for python) to check if a file is changed. For other programming languages see slide 26 of the talk " GitOps and Secrets:State of the Union “. The presenter, Kostis Kapelonis, also created a simple repository to show how this works.

I implemented his solution in my own environment as well.

03 - Refresh secrets AWS

With my background in AWS, I liked to know how this worked with secrets in AWS. I liked the idea of automated change of secrets a lot, so I woundered if something like this would exist for parameters as well. I would like to get an operator that “translates” AWS SSM parameters to ConfigMaps. Unfortunately this doesn’t exist (yet), but you can use External Secrets Operator to translate SSM parameters into Kubernetes secrets. Which might be somewhat confusing.

In the example repository you will find both examples for External Secrets Operator connections with AWS Secrets Manager and with SSM Parameter Store.

04 - Crossplane

Learning Crossplane

The External Secrets Operator can be used to get secrets and parameters from AWS into Kubernetes. But it can be done the other direction as well: we can add SSM parameters and secrets from Kubernetes to AWS. This might be done via Crossplane. I saw the video on the crossplane website before the conference, but didn’t have time to look into it further.

The talk on the conference (” Crossplane - The Cloud Native Framework for Platform Enginering" then went a little bit too fast for me… When I was home I took more time to look into it more properly: I first followed the demos on the Crossplane site and put the files in my example repository.

Creating SSM parameters and secrets

When I created crossplane objects for an SSM parameter and a secret, there are some differences in the way Crossplane deals with these objects. The way that Crossplane refers to AWS objects is also different: for SSM parameters this works via annotations:

apiVersion: ssm.aws.m.upbound.io/v1beta1
kind: Parameter
metadata:
  name: crossplane-demo-parameter
  annotations:
    crossplane.io/external-name: /kubecon26/examples/04-crossplane/demo-parameter
spec:
  forProvider:
    type: String
    insecureValue: hello-from-crossplane
    region: eu-west-1

For secrets, this is via a (Kubernetes) secret name parameter in forProvider:

apiVersion: v1
kind: Secret
metadata:
  name: crossplane-demo-secret-value
  namespace: default
type: Opaque
stringData:
  secret-string: '{"demo":"hello-from-crossplane"}'
---
apiVersion: secretsmanager.aws.m.upbound.io/v1beta1
kind: Secret
metadata:
  name: crossplane-demo-secret
spec:
  forProvider:
    name: kubecon26/examples/04-crossplane/demo-secret
    description: Demo secret created by Crossplane
    region: eu-west-1
---
apiVersion: secretsmanager.aws.m.upbound.io/v1beta1
kind: SecretVersion
metadata:
  name: crossplane-demo-secret-version
spec:
  forProvider:
    region: eu-west-1
    secretIdRef:
      name: crossplane-demo-secret
    secretStringSecretRef:
      name: crossplane-demo-secret-value
      key: secret-string

This leads to:

It would be better to use one way-of-working for this within Crossplane.

When you use the newest versions (on the time of writing this blog), it is not possible to put values in SecureStrings in SSM parameters or in SecretValues in secretsmanager. These have to be pulled from a kubernetes secret. The Kubernetes secret for the SSM parameter can be put in the namespace of the application, the secret for secretsmanager should be in the default namespace (which is not what one would expect when one deploys the solution).

When you delete a secret and then recreate it, you might get an error message that says:

failed to create the resource: [...] creating Secrets Manager Secret
(kubecon26/examples/04-crossplane/demo-secret): operation error Secrets
Manager: CreateSecret, https response error StatusCode: 400, RequestID:
1ab23456-7890-1cde-fgh2-34i5678jkl9m, InvalidRequestException: You can't
create this secret because a secret with this name is already scheduled
for deletion.

This can be solved by using the AWS CLI:

aws secretsmanager delete-secret \
  --secret-id kubecon26/examples/04-crossplane/demo-secret \
  --force-delete-without-recovery \
  --region eu-west-1 \
  --profile your-profile

Crossplane CLI

I then tried to follow along with the Kubecon demo in this repository, but the PR was already merged and some commands were different than those in the repository. I could follow along using the following commands on the control node of my AWS kubecon26-example-repo:

Setup

# This commands are done for you
cd /clone
git clone https://github.com/crossplane/cli

cd /clone/cli
go install ./cmd/crossplane
PATH=$PATH:/home/kubernetes/go/bin

Demo Script

You can follow along with the rest of the demo, executing

crossplane project init hello-amsterdam && cd hello-amsterdam

from the ~/kubernetes directory.

In step 7, use v1.34.0 instead:

crossplane dependency add k8s:v1.34.0
echo
echo "Here's what crossplane-project.yaml looks like after adding the dependency:"
echo
cat crossplane-project.yaml

In step 10, don’t use the –crossplane-version argument:

crossplane composition render --timeout=10m examples/webapp/podinfo.yaml apis/webapps/composition.yaml

In step 14, use curl instead of open:

kubectl port-forward svc/$(kubectl get svc -l crossplane.io/composite=podinfo -o jsonpath='{.items[0].metadata.name}') 9898 >/dev/null 2>&1 &
curl http://localhost:9898

It was very nice to see that using crossplane is made easier by just having to add the relevant code and let the crossplane CLI do the rest.

05 - OpenTelemetry

In Kubernetes services can be slow, which can be cause by calls to the DNS. I liked the talk about " DNS Tracing and Metrics Via eBPF in OpenTelemetry “. Sometimes the slowness is not caused by the DNS server, but by the DNS clients: maybe by not using the FQDN of the server. But where does this happen?

We can find out by using OpenTelemetry eBPF Instrumentation (OBI). Both metrics and spans are supported. I added the configuration for eBPF and OpenTelemetry in the example repository.

You can look at the dashboard DNS Requests (eBPF) for DNS requests within this cluster.

I was a little bit surprised that the DNS requests were not visible in the same way as in the Kubecon presentation. The reason was, that both Python and (the current implementation of) go have libraries that add search paths to the URL when no dots are present. The first search path to add is 05-opentelemetry.svc.cluster.local, which is the correct one for my service.

I changed the script a little bit to add different paths. This is the result:

06 - Policy Engines

One of the nice things of Kubecon is that some presentors give overviews of different solutions for the same problem. The presentation Policy Engines for Kubernetes: Picking One Without Losing Your Mind by Naburan Pal did just that. He did a great job of showing code, advantages and disadvantages of four policy engines: ValidatingAdmissionPolicies, Kyverno, OPA/Gatekeeper and Kubewarden.

I implemented three of four in my example repository: Kubewarden is used less in the community and has a different architecture so I left this one out.

Conclusion

It helped me to play with knowledge I learned from conferences like Kubecon. You can do as well, by using the example repository.

Have fun!

The European Sovereign Cloud, more complex than it seems?

Wed, 11 Mar 2026 00:00:00 +0000

Introduction

A few months ago, AWS introduced its AWS European Sovereign Cloud. It is delivered in the middle of a broader discussion about sovereignty, where people seem to have very strong opinions: according to some people there is just one option and that is to leave the American cloud providers as fast as possible. Other people just wait and see. Or don’t see any risks for their organization. How to deal with these differences? Before making this decision, organizations should weigh factors that are rarely discussed publicly.

Risks

Risk is based on both the chance that an event occurs and on the impact. When you run your primary workloads in an American cloud, it can be frightening that the US administration might read your data or stop your resources. Or make your workloads inaccessible by, for example, shutting down network devices. The big question is here: what should we defend ourselves against? What is real to expect and what isn’t?

Is it realistic to expect the US administration to shut down the access to a whole region? Or will they limit themselves to the resources in individual accounts/tenants? Is the data protected (enough) by the limitations that the hyperscalers put upon themselves, or can we trust no-one when it comes to our data?

When you listen to some (Dutch) politicians, there is just one solution for (Dutch) governmental organizations and that is to move out of the American clouds and go to European clouds. That sounds safe, because the Cloud Act is not valid in those clouds, so this seems to be more secure than the American cloud providers.

But is this the whole story? The hyperscalers have services in place that defend users in the sovereign cloud against for example DDoS attacks. Via honeypots they also figure out how attackers attack systems on the internet. When an attacker tries to connect to their command-and-control server then the access from any workload on that hyperscaler cloud to that hacker is made impossible. Do European cloud providers have these security measures in place as well? And, when they do, is the chance of hitting one of the European Sovereign Cloud honeypots as high as hitting one of the honeypots of the hyperscaler clouds? The idea that European Sovereign Clouds are by default more secure than non-European Sovereign Clouds might not be true…

Another aspect that is often overlooked is that political uncertainty is not limited to countries outside Europe. It also exists within EU member states themselves. In several European countries, parties with strong views on national identity or a more populist, inward‑looking agenda are gaining influence. This raises a relevant question: what happens when such parties become part of a government and introduce policies that affect digital infrastructure, data governance, or international cooperation? A migration to a European Sovereign Cloud may feel like the right step today, but future political developments could still require organizations to rethink their choices. It is therefore important to acknowledge that no cloud strategy is entirely insulated from political dynamics, inside or outside the European Union.

Functionality in the different clouds

Even when one would migrate to the AWS European Sovereign Cloud from the current AWS global cloud, there are drawbacks when you compare them to the global cloud. New functionality will become available later than in the global infrastructure cloud and some functionality is not present (yet) or works (slightly) differently.

One of the disadvantages of the European Sovereign Clouds is that their maturity is less than those of the hyperscaler clouds: hyperscaler clouds have a lot of functionality built in their services that European Sovereign Clouds just don’t have. That means that in some cases you have to implement these features yourself. Both developing and maintaining this functionality costs time, time that you cannot spend on developing your business applications.

One example of this is AWS Secrets Manager: it can be configured to change passwords in Secrets Manager and then change the same password in for example your MySQL database. Setting this up can be done in minutes. Implementing this functionality in one of the European Government Clouds will cost time.

Costs of migration

That brings me to the cost of migration. As with any migration, migrating to the European Sovereign Cloud costs time and money. It takes time to choose the most appropriate European Sovereign Cloud for your use case. It takes time to prepare the migration, test it in test environments, it takes capacity in the organization that cannot be used to deliver new functionality or help customers.

Network and identity

When you want to be fully independent of the USA Government, you have to look into the network and identity providers as well. Many companies rely on Microsoft, either via Active Directory or via EntraID. EntraID is well secured and has a lot of functionality that isn’t present in other Identity and Access Management systems.

From a networking perspective, your organization might use American companies like Equinix to connect from your on-premises environment to your cloud provider.

When you want to be fully independent of American companies you should search for an alternative solution here as well. In some cases (for example the AWS European Sovereign Cloud) you might have to arrange a direct connection to another country instead of being able to use a connection within your own country.

Direct Connect locations for AWS European Sovereign Cloud

Direct Connect locations for T Cloud Public

Emotions

When you listen to pro and cons of the European Sovereign Cloud, you hear a lot of emotions. Anger and fear for the US Government. Anger at ourselves as European organizations, that we didn’t choose for European Cloud providers from the start.

Last month I tried to use a more rational approach, looking at pros and cons. Looking at all kind of risks. But in the end, I had to admit that I also feel emotions in these discussions. In my case I felt sadness, that I have to put effort in learning about clouds that are less mature from a technical point of view. It’s my job and I will do it – but it might influence my point of view.

Maybe we can agree that it’s just impossible to have a rational discussion on this topic, just because one cannot determine the probability of any of the risks?

Conclusion

When you listen to some evangelists of the European Sovereign Cloud, it’s all very easy. The choice to migrate to the European Sovereign Cloud is easy – because the impact of a decision of the US Government can destroy your organization. The disadvantages of these choices are rarely mentioned. I think that for some organizations it’s good to migrate to the European Sovereign Cloud. For others, it just doesn’t make sense. It takes time and money to investigate what is the best for your organization. From where I stand, the discussion needs more reasoning and less emotions - we should at least try not to look just at the impact but also look at the probability of that risk and the costs and the disadvantages of a decision before making that decision. And then agree that there really is no silver bullet in this discussion, take a decision and live with the consequences.

===
Image by kp yamu Jayanath from Pixabay. I added the European stars via Claude.

AWS Nuke with AWS StepFunctions

Sun, 28 Dec 2025 00:00:00 +0000

A client of mine was updating all Python runtime versions to the newest version. They also created their own cleanup Lambda functions for development resources. This just contained a specific set of AWS resource types. I advised them to use AWS Nuke, as this can cleanup way more AWS resources. I volunteered to create a generic solution for this, that also tries to get the newest aws-nuke version when it is released. You can find that solution in this github repository

Installation

I tried to keep the installation as simple as possible: it’s just a matter of copying a template file with environment variables to your own version of it, then change the variables in the way you want to use the tool and go. Some settings that you can use are:

Schedule

Manual

No administrator will use a tool like aws-nuke without first wanting to see what resources will be deleted. You can use the manual mode for this: set the SCHEDULE variable to manual, configure the rest of the setenv.sh file and then start the deployment with bash ./scripts/deploy-cdk.sh. When the deployment is ready, you can start the execution of the dry run via ./scripts/start-manual-workflow.sh. When the run is ready you will receive an email, you can then decide to tag some resources that you want to keep or to go ahead and remove the resources in the list. This can be done using the ./scripts/approve-execution.sh script. After the removal of the resources you will get an email with the link to the output of aws-nuke.

Cron expression

You can also add a cron expression, for example cron(0 4 ? * WED *) to run every Wednesday at 4:00 in the morning. After the deployment with bash ./scripts/deploy-cdk.sh, the tool will start on the specified moment. You can use the AWS Cron Expression Validator & Generator to create valid AWS cron expressions if you need to.

You will not get an email for these executions (the aws-nuke logs are stored in the S3 bucket). It will also directly remove the resources, without doing a dry run first.

When you want to do a manual cleanup in between, you can still use the start-manual-workflow.sh and approve-execution.sh scripts. These scripts will still send you emails.

Blocklist accounts

AWS Nuke will never run in blocklist accounts. It might be useful to fill in the number(s) of your production account(s) here.

Nuke version and enforce version

Some people (like me) like tools like this to always use the most recent version of aws-nuke. In that case, you can fill in the most recent version number from the aws-nuke releases page in NUKE_VERSION and keep the ENFORCE_VERSION on false. The tool will then try to get the most recent version and, if it doesn’t succeed in determining which version that is, fall back to the version number in NUKE_VERSION.

When you, on the other hand, always want to use a specific version from aws-nuke because you have an extended test process in place outside this tool, you can put the version number that you want to use in the NUKE_VERSION variable and put the ENFORCE_VERSION on true.

Architecture

My CDK script is based on an AWS Step Function to first create a configuration script for AWS Nuke and then start the execution.

The first Lambda function will create the config file, then the second Lambda function starts AWS Nuke. This is either in dry-run mode or in execution mode - the difference is based on the parameters of the workflow. Another parameter is used to check if a notification should be sent.

The configuration file and the output are stored in an S3 bucket, a lifecycle policy on this bucket will remove the files after 30 days (which is configurable). The scheduling is done by EventBridge rules. SNS will send the email. From an AWS perspective this solution is pretty straightforward.

Config file

The configfile will keep certain resources alone:

Of course, the resources that are tagged with the key/value pair that we provided
Resources related to AWS Control Tower
Resources related to the CDK Toolkit
Resources related to our project

When I initially ran aws-nuke, I saw a lot of AWS errors from resources that didn’t exist anymore. I included all these resource types in the exclusion list.

The configfile for aws-nuke that is created based by the Lambda function looks like:

blocklist:
- '123456789012'
regions:
- eu-west-1
- eu-central-1
resource-types:
  excludes:
  # Network interfaces and attachments (removed with parent resources)
  - EC2NetworkInterface
  - EC2DHCPOption
  - EC2InternetGatewayAttachment

  # Bedrock issues
  - BedrockModelCustomizationJob

  # Deprecated/unused resource types
  - CloudSearchDomain
  - CodeStarProject
  - ElasticTranscoder*
  - FMSNotificationChannel
  - FMSPolicy
  - OpsWorks*
  - QLDBLedger
  - Lex*
  - MachineLearning*
  - RoboMaker*
  - ShieldProtection*
  - AWS::Timestream::*
accounts:
  999999999999:
    filters:
      __global__:
      - property: tag:Cleanup
        value: persist
      - property: tag:aws:cloudformation:stack-name
        value: CDKToolkit
      - property: tag:aws:cloudformation:stack-name
        type: glob
        value: StackSet-AWSControlTowerBP-*
      - property: Name
        type: glob
        value: aws-controltower-*
      - property: Name
        type: glob
        value: AWSControlTower*
      - property: Name
        type: glob
        value: aws-nuke*
      CloudFormationStack:
      - property: Name
        type: glob
        value: StackSet-AWSControlTowerBP-*
      CloudWatchLogsLogGroup:
      - property: Name
        type: glob
        value: /aws/lambda/aws-nuke-*
      - property: Name
        type: glob
        value: /aws/lambda/aws-controltower-*
      S3Object:
      - property: Bucket
        value: aws-nuke-aws-nuke-bucket-999999999999
      - property: Bucket
        type: glob
        value: cdk-hnb659fds-*
      SNSSubscription:
      - property: TopicARN
        type: glob
        value: arn:aws:sns:*:999999999999:aws-nuke-*
      - property: TopicARN
        type: glob
        value: arn:aws:sns:*:999999999999:aws-controltower-*
      SNSTopic:
      - property: TopicARN
        type: glob
        value: arn:aws:sns:*:999999999999:aws-controltower-*

Warning on volumes of EC2 instances

Tag based protection works fine when everything that should be protected is tagged correctly. In general tags that are applied to a CloudFormation stack are also applied on the resources that are created by the stack. This is also true for EC2 instances, but volumes that are created by the EC2 instances are not tagged by default. You can do that by adding the PropagateTagsToVolumeOnCreation: true property to the EC2 instance.

When you don’t add this tag to the volume, no tags will be attached to the volume and aws-nuke will try to remove the volume. But because it is attached to the EC2 instance, the volume cannot be removed. After a few times aws-nuke will stop trying to remove the volume. Though the EC2 instance will run like before, these retries cost time. And because Lambda functions are billed based on time this will also cost some money.

Kiro

I created this solution with Kiro, which saved me about 50% of the time. Of course I checked all the code before releasing it via my github repository.

There are advantages and disadvantages on using Kiro. I didn’t like it that Kiros code sometimes didn’t work, even though Kiro in general needs less tries between the first try and the working project as I would need.

Kiros initial idea was to create a step function that would first start a dry run of aws-nuke and then wait for manual approval. More or less like this simplified example . It tried at least 15 times before giving up and choosing the current solution, where the same workflow is started for both the dry run and the final execution. In the end, I didn’t bother: the solution works and most people will use the scheduled variant more than the manual way of working anyway.

Kiros solution also costs me more than one day to clean up all the documentation files, extra checks and the extra configuration that Kiro added - just because it wasn’t sure what keywords to use. It put simply two different keywords with the same configuration in the configuration file.

In some cases, Kiro put all the code in one Lambda function. Some code was put there multiple times, rewriting this code improved the readability.

There was also a bug in fsWrite, which is Kiros way of writing files to the operating system (in my case Fedora). Kiro saw the content of the files, where the content wasn’t visible from the operating system. This lead to a lot of errors in cdk runs that were not necessary. It was simple to solve: you can create the file .kiro/agents/default.json and put the following content in it:

{
  "description": "Default agent with fs_write workaround",
  "allowedTools": [
    "fs_read",
    "execute_bash"
  ],
  "toolsSettings": {
    "execute_bash": {
      "autoAllowReadonly": true
    }
  },
  "instructions": "For file creation and modification, always use
  execute_bash with shell commands like 'cat > file << EOF' or
  'echo content > file' instead of fs_write tool due to zero-byte
  file issues."
}

Kiro was able to give me this solution - I just had to ask for it…

Image

Image by Reginaldo Martins from Pixabay

Mini Dynatrace Newsletter - November 2025 Edition

Tue, 18 Nov 2025 00:00:00 +0000

I will try to share and highlight new relevant Dynatrace features (mostly from the Dynatrace release notes) more frequently - seasoned with my personal opinions, ideas and practical context. So, welcome to my new and very first Mini Dynatrace Newsletter!

Settings - Holiday-aware baseline modification

When? available since 21 oct 2025, SaaS version 1.326 (auto-update, so if you have SaaS this feature is available automatically) Summary (from the release notes): “Public holidays have a significant impact on the usage behavior of services and applications. These deviations also affect the accuracy of the service and application baseline. To minimize the risk of false positive alerts, specific public holidays are systematically excluded from the service baseline and application baseline. Excluded public holidays are: New Year, Easter, Thanksgiving, Black Friday and Christmas. This setting is enabled by default. To disable the feature, go to Settings > Anomaly Detection> Holiday-aware baseline modification.” Details: The holiday-aware baseline modification now is a default setting - applied immediately after the release of the SaaS 1.326 version (yes, you and/or your customers are already using it in case you’re on SaaS). This means, that during holidays, there won’t be a baseline created for any metric - nevertheless, the data will still be stored, just not as the new reference baseline. A week after the holiday, the holiday will not be used as a reference point for performance. The logic slightly resembles the logic of the “Frequent Issue Detection”, as described here: Dynatrace Docs - Detection of Frequent Issues My 2cts: A step in the right direction - and something that many customers asked for. Unfortunately focused on the American market (holidays in America, not the Dutch holidays and no fine-tuning or holiday-definition possible). So far, the settings also seem to be mostly focused on “retail” customers. In the future, I would love to see this feature improved with individual holiday definitions. The users KIKON and Anton Pineiro - DynaMight on the Dynatrace community - have created a product idea already. If you have a similar view as me and them on the new feature, feel free to upvote and support: Dynatrace Community - Define distinct type of days for Holidays-aware baseline

DQL/Dashboards/Notebooks - Visualize data relationships with the new Scatterplots

When? since 21 oct 2025, SaaS version 1.326 (auto-update, so if you have SaaS this feature is available automatically) Summary (from the release notes): “You can now use scatterplots in Dashboards and Notebooks to visualize data relationships and identify patterns, such as correlations between response time and request count. This new visualization helps you analyze metrics more effectively and uncover insights in your observability data.” Mini Demo: Let us start off easy with a scatterplot of one metric (y-axis) against the timeframe (x-axis), so a metric over time. This is a nice alternative to line graphs.

timeseries avg(dt.service.request.response_time), by:{dt.entity.service}
|fieldsAdd entityName(dt.entity.service)
|filter dt.entity.service.name == "CreditCardValidation"

If I select a bigger timeframe now, Dynatrace will adjust the interval, meaning that many datapoints will be merged to one. I am not keen on that, as it defeats the purpose of a scatterplot which is supposed to show all datapoints, but we can avoid it by adding an interval ourselves Like that:

timeseries avg(dt.service.request.response_time), interval:1m, by:{dt.entity.service}
|fieldsAdd entityName(dt.entity.service)
|filter dt.entity.service.name == "CreditCardValidation"

Now let’s get started on the interesting part, with two different “metric variables” on the x-axis and the y-axis. A small warning/disclaimer: If you create two metrics as a line chart on a dashboard, you can actually also just append them - less beautiful maybe, but it works. As we do want to know the exact y-value for every x-value, to understand their relationship (and these values will be shown without their timestamp on the dashboard), this approach will not work (at least not without additional data engineering with DQL). So you have to use join or lookup commands or create two timeseries immediately in the first step (see my example). More information on joins and lookups: Dynatrace Docs - DQL Correlation and join commands

If you want to, you can use the Dynatrace playground to test the queries: Dynatrace Playground

First, we create two metric timeseries:

timeseries {response_time_p99 = percentile(dt.service.request.response_time, 99),
request_count = sum(dt.service.request.count)},
interval: 1m, by:{dt.entity.service}, union:true
|fieldsAdd entityName(dt.entity.service)
|filter dt.entity.service == "SERVICE-1234567891011"
//enter your own service ID here

Unfortunately, we cannot choose the scatterplot visualization immediately. I actually would have liked that… so we will go for a workaround to make it possible. This is not documented in the release notes or in the Dynatrace Documentation so feel free to follow my steps.

We need to look at a record list, and no longer at two timeseries. The following commands will convert the timeseries to single records:

// Expanding timeseries to single records
|fields all = iCollectArray(record(response_time_p99= response_time_p99[],
request_count = request_count[])),dt.entity.service.name
|expand all
|fieldsAdd request_count = all[request_count], response_time_p99 = all[response_time_p99]
|fieldsRemove all

Result:

Now it makes sense to remove the outliers (a boxplot would be great now to know which ones to exclude, but that feature does not exist in Dynatrace yet). So, we add the following rules (based on the visual impression of outliers):

//Removing outliers
|filterOut request_count > 150 //(number of requests)
|filterOut response_time_p99 > 100000 //(100ms)

Result:

The amount of requests per minute seems to have a positive relationship with the response time of the slowest 1% requests (more requests are related to a higher response time).

Let’s check if the visual impression of a positive correlation between the “slowest 1% requests” response time and the request count actually exists. For that, we will add the |summarize correlation() command.

//Pearson correlation coefficient
|summarize correlation(request_count,response_time_p99)

The result is indeed a positive relationship:

|summarize correlation() already exists for some time, but does fit the use case here, that’s why I included it in the newsletter. My 2cts: Mostly happy. More dashboarding options and more data science is always a good step forward. I do miss the option to add a trendline (see the example with R).

For me, this belongs to a good scatterplot. Unfortunately, there is also no documentation page for scatterplots so far, so the link from the DQL tile does not work yet:

Furthermore, it is not possible yet to simply add two timeseries and turn them into a scatterplot with a normal metrics tile or DQL tile without intermediate steps (see the error message in this newsletter). That could be improved in the future to make this feature more accessible to everyone.

Okay, why would I call this a mini newsletter? I mean, this was quite a long article…? This was only a sub-set of all the new features published in the last weeks. In the release notes, you can find even more features, but mostly with only a few details and little description. With these types of newsletters, I am trying to give these improvements and adjustments a bit of practical context and give everyone a chance to test them as soon as possible.

Azure Monitor Alerts and Dynatrace

Fri, 29 Aug 2025 00:00:00 +0000

Introduction

Not too long ago, we were asked to send Azure Monitor Alerts to Dynatrace. It didn’t seem too difficult, there even was a page in the Dynatrace documentation how to do this [1]. When we followed this document, the result was somewhat disappointing: monitor alerts were sent to Dynatrace, but the events that were created didn’t contain many properties:

{
  "records": [
    {
      "timestamp": "2025-08-15T10:24:47.351000000+02:00",
      "event.id": "1234567890123456789_0123456789012",
      "event.status": "CLOSED",
      "dt.davis.timeout": "15",
      "source": "Azure activity log - Administrative",
      "affected_entity_types": [
        "dt.entity.custom_device"
      ],
      "event.provider": "ONEAGENT",
      "event.category": "INFO",
      "affected_entity_ids": [
        "CUSTOM_DEVICE-D1234A56B78C9012"
      ],
      "dt.davis.mute.status": "NOT_MUTED",
      "event.status_transition": "CREATED",
      "annotation_type": "Informational",
      "dt.entity.custom_device": "CUSTOM_DEVICE-D1234A56B78C9012",
      "event.group_label": "Custom annotation",
      "dt.source_entity": "CUSTOM_DEVICE-D1234A56B78C9012",
      "maintenance.is_under_maintenance": false,
      "dt.source_entity.type": "custom_device",
      "dt.openpipeline.source": "oneagent",
      "event.name": "Annotation",
      "event.kind": "DAVIS_EVENT",
      "event.start": "2025-08-15T10:17:22.075000000+02:00",
      "dt.davis.is_frequent_event": false,
      "event.description": "Microsoft.DBforMySQL/flexibleServers/write: Started (Informational)",
      "event.end": "2025-08-15T10:39:47.349000000+02:00",
      "dt.davis.impact_level": "Infrastructure",
      "event.type": "CUSTOM_ANNOTATION"
    }
  ]
}

The name of the alert rule wasn’t present, there was no name of the resource or the resource group. When I tried to get the custom device name via entityName(dt.entity.custom_device), it came back with null. Our Azure environment has hundreds of alerts, how can one make the difference between them and how to decide when a problem should be created - and when not to do so? We couldn’t use this event to determine the name of the application.

The question was then: was this Dynatrace not showing enough data, or is it Azure not sending the data to Dynatrace? My colleague pointed to the nice (free!) website where one can test webhooks: https://beeceptor.com. Via this website we got the complete text that is sent to Dynatrace by the webhook in Azure Monitor Alerts:

 {
    "schemaId":"Microsoft.Insights/activityLogs",
    "data":{
        "status":"Activated",
        "context":{
            "activityLog":{
                "authorization":{
                    "action":"Microsoft.DBforMySQL/flexibleServers/write",
                    "scope":"/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij1/resourceGroups/my-resourcegroup/providers/Microsoft.DBforMySQL/flexibleServers/my-database"
                },
                "channels":"Operation",
                "claims":"{...}",
                "caller":"frederique.retsema@conclusionxforce.nl",
                "correlationId":"1ab2345c-d678-901e-234f-5678gh90123i",
                "description":"",
                "eventSource":"Administrative",
                "eventTimestamp":"2025-08-13T10:56:14.2468371+00:00",
                "httpRequest":"",
                "eventDataId":"a123b456-c78d-9012-3456-ef7gh8ij90kl",
                "level":"Informational",
                "operationName":"Microsoft.DBforMySQL/flexibleServers/write",
                "operationId":"a12b3456-c7d8-90e1-fg23-45hijkl6789m",
                "properties":{
                    "eventCategory":"Administrative",
                    "entity":"/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/resourcegroups/my-resourcegroup/providers/Microsoft.DBforMySQL/flexibleServers/my-database",
                    "message":"Microsoft.DBforMySQL/flexibleServers/write",
                    "hierarchy":"12345ab6-cdef-78gh-i9jk-0123l4m56789/123a45b6-78c9-0d12-e3f4-567gh8901ij2"
                },
                "resourceId":"/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/resourcegroups/my-resourcegroup/providers/Microsoft.DBforMySQL/flexibleServers/my-database",
                "resourceGroupName":"my-resourcegroup",
                "resourceProviderName":"Microsoft.DBforMySQL",
                "status":"Succeeded",
                "subStatus":"",
                "subscriptionId":"123a45b6-78c9-0d12-e3f4-567gh8901ij2",
                "tenantId":"12345ab6-cdef-78gh-i9jk-0123l4m56789",
                "submissionTimestamp":"2025-08-13T10:58:59+00:00",
                "ReceivedTime":"2025-08-13T10:58:59.3078161+00:00",
                "ingestionTime":"2025-08-13T10:59:02.8559575+00:00",
                "resourceType":"Microsoft.DBforMySQL/flexibleServers"
            }
        },
        "properties":{}
    }
}

This contains more properties: when we would have this in Dynatrace, we could check on the resource, the resource group, maybe even the tenant ID to base our decisions on. We still wouldn’t have the name of the alert, though. Or the tags of the alert rule. To have the name of the alert and the tags as well would make our lives easier, because that makes troubleshooting a lot easier: we could then connect a problem in Dynatrace to an alert in Azure. So how would we get this?

Different route

So, the original route goes from the Azure Monitor Alert Rule, via an Action Group to the dedicated Azure Web Hook within ActiveGate in Dynatrace (https://<YOUR_ACTIVEGATE_ADDRESS>:9999/modules/azure_monitoring/alerts_webhook?token=<YOUR_API_TOKEN>).

When we add an Azure Function in between and then let the Azure Function use the Dynatrace log ingest API, we are free to put any properties we want in the call.

This works (see the repository with Terraform code here: [2]), now we have all the information we need, including the name of the alert rule:

fetch logs
| filter log.source == "Azure Monitoring Alert"
| sort timestamp desc
| limit 1

This leads to f.e. the following inserted log item:

    {
      "timestamp": "2025-08-15T18:44:45.896000000+02:00",
      "content": "...",
      "event.type": "LOG",
      "log.source": "Azure Monitoring Alert",
      "loglevel": "DEBUG",
      "status": "INFO",
      "data.alertcontext.activity log event description": "",
      "data.alertcontext.authorization.action": "Microsoft.DBforMySQL/flexibleServers/write",
      "data.alertcontext.authorization.scope": "/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/resourceGroups/my-resourcegroup/providers/Microsoft.DBforMySQL/flexibleServers/my-database",
      "data.alertcontext.caller": "frederique.retsema@conclusionxforce.nl",
      "data.alertcontext.channels": "Operation",
      "data.alertcontext.claims": "{...}",
      "data.alertcontext.correlationid": "12345678-9a0b-1cd2-e34f-5g678901g234",
      "data.alertcontext.eventdataid": "a123bc4d-e5f6-789g-0h12-345i678j9012",
      "data.alertcontext.eventsource": "Administrative",
      "data.alertcontext.eventtimestamp": "2025-08-15T16:40:01.8435309+00:00",
      "data.alertcontext.httprequest": "",
      "data.alertcontext.ingestiontime": "2025-08-15T16:42:46.8093833+00:00",
      "data.alertcontext.level": "Informational",
      "data.alertcontext.operationid": "ab1c2345-6d78-90ef-1g2h-34ij567k8lm9",
      "data.alertcontext.operationname": "Microsoft.DBforMySQL/flexibleServers/write",
      "data.alertcontext.properties.entity": "/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/resourcegroups/my-resourcegroup/providers/Microsoft.DBforMySQL/flexibleServers/my-database",
      "data.alertcontext.properties.eventcategory": "Administrative",
      "data.alertcontext.properties.hierarchy": "12345ab6-cdef-78gh-i9jk-0123l4m56789/123a45b6-78c9-0d12-e3f4-567gh8901ij2",
      "data.alertcontext.properties.message": "Microsoft.DBforMySQL/flexibleServers/write",
      "data.alertcontext.receivedtime": "2025-08-15T16:42:44.7210637+00:00",
      "data.alertcontext.status": "Succeeded",
      "data.alertcontext.submissiontimestamp": "2025-08-15T16:42:44+00:00",
      "data.alertcontext.substatus": "",
      "data.alertcontext.tenantid": "12345ab6-cdef-78gh-i9jk-0123l4m56789",
      "data.essentials.alertcontextversion": "1.0",
      "data.essentials.alertid": "/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/providers/Microsoft.AlertsManagement/alerts/1a2345b1-6cd7-8e90-1f2g-3h4i5678j901",
      "data.essentials.alertrule": "Alert for creation of database",
      "data.essentials.alertruleid": "/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/resourceGroups/my-resourcegroup/providers/microsoft.insights/activityLogAlerts/Alert for creation of database",
      "data.essentials.alerttargetids": [
        "/subscriptions/123a45b6-78c9-0d12-e3f4-567gh8901ij2/resourcegroups/my-resourcegroup/providers/microsoft.dbformysql/flexibleservers/my-database"
      ],
      "data.essentials.configurationitems": [
        "my-database"
      ],
      "data.essentials.description": "Testing Dynatrace",
      "data.essentials.essentialsversion": "1.0",
      "data.essentials.fireddatetime": "2025-08-15T16:44:45.8960322Z",
      "data.essentials.investigationlink": "https://portal.azure.com/#view/Microsoft_Azure_Monitoring_Alerts/Issue.ReactView/alertId/%2fsubscriptions%123a45b6-78c9-0d12-e3f4-567gh8901ij2%2fresourceGroups%2fmy-resourcegroup%2fproviders%2fMicrosoft.AlertsManagement%2falerts%2f1a2345b1-6cd7-8e90-1f2g-3h4i5678j901",
      "data.essentials.monitorcondition": "Fired",
      "data.essentials.monitoringservice": "Activity Log - Administrative",
      "data.essentials.originalertid": "a123bc4d-e5f6-789g-0h67-890i123j4567_8k9l012m3456n7o89pqr0st123uvw45x",
      "data.essentials.severity": "Sev4",
      "data.essentials.signaltype": "Activity Log",
      "data.essentials.targetresourcegroup": "my-resourcegroup",
      "data.essentials.targetresourcetype": "microsoft.dbformysql/flexibleservers",
      "dt.auth.origin": "dt0x01.ABCDEFG12H3IJKLMNOPQRSTU",
      "dt.openpipeline.pipelines": [
        "logs:default"
      ],
      "dt.openpipeline.source": "/api/v2/logs/ingest",
      "schemaid": "azureMonitorCommonAlertSchema"
    }

By making it a log line instead of an event, it is possible to use OpenPipeline to create metrics and Events/Buziness Events out of this - as we can determine in the future. The metrics can then be used in Davis Anomaly Detectors, which can also use the severity in the log line.

Adding tags

Both the original webhook calls that are sent to Dynatrace and the calls to Azure Function don’t send the tags of the monitor alert to Dynatrace. This is a pitty, because the tags can be used to add the application name, or the business unit name, or the security rating of the system to Dynatrace. You don’t want to maintain this kind of information in multiple places. Therefore, the solution should be changed a little bit:

Now, the lines with tags in the log line become for example:

{
    ...
      "alert.tag.goal": "Test Dynatrace",
      "alert.tag.name": "Frederique",
      "alert.tag.tagwithspaces": "Spacy",
      "resourcegroup.tag.goal": "ResourceGroup Test Dynatrace"
    ...
}

As you can see here, spaces in the original tag are removed (the original tag key was Tag with spaces)

Security

To allow the Azure Function to read the Alert Rules, a managed identity is created. The Dynatrace URL and the Dynatrace token are both stored in an Azure Key Vault.

Reusable

This solution is very generic. Use it in any environment you like.

Installation

You can find the Terraform code in my repository [2]. First, copy the file setenv.template.sh to setenv.sh and change the content. Likewise, copy the file terraform.auto.template.tfvars to terraform.auto.tfvars and change the contents.

Then, use the following commands to deploy this solution to your environment:

. ./setup.sh
. ./login.sh
. ./deploy.sh

If you ever want to remove the resources from your environment, use the following commands to do so:

. ./setup.sh
. ./remove.sh

Permissions

When you look in the main.tf terraform file, it might seem double to both assign “Monitoring Reader” and “Reader” permissions. When you don’t mind to give reader permissions to the Azure Function, then remove the “Monitoring Reader” permissions. When you do mind, then remove the “Reader” permissions. When you only give “Monitoring Reader” permissions, the Azure Function will not send back tags of the resource group of the resource that triggert the alert rule.

Testing

You can use the test script to send a record with the most basic information to the Azure Function. The steps are:

Copy the testfunction.template.sh file to testfunction.sh
Copy the default key from the Azure Function App (open the function app, left menu, Functions > App keys > default) and paste it in testfunction.sh (in the ?code=… part of the URL)
Change the other variables in this script as well

The curl will result in a 204 errorcode when the call is successful. You can also look in the invocation tab of the Azure Function.

Configuration

Now the Azure Function works, you can add the Azure Function to the action group of the monitoring alert. Example:

Links

[1] Dynatrace documentation how to send Azure Monitor Alerts to Dynatrace: see here [2] Repo with Terraform code: https://github.com/FrederiqueRetsema/azure-alerts-dynatrace

Pro-active observability on top of reactive observability, with predictive AI (and workflows) in Dynatrace

Mon, 21 Jul 2025 00:00:00 +0000

Artificial Intelligence (AI) is a very contemporary topic – also for Observability. While generative AI can probably be seen as the most-discussed topic right now, I have decided to focus on a great and powerful classic. The symbiosis of predictive AI, observability and automation.

Ref: https://www.dynatrace.com/platform/artificial-intelligence/

At one of my current assignments, forecast tiles have been added as an essential element to all application dashboards in Dynatrace. Infrastructure forecasting, including memory usage, CPU usage and disk space has become a very easy way for them to avoid larger issues, for which there have been clearly noticeable signs. There are multiple apps in Dynatrace, which can be used to create the metric forecasts. The dashboards app and the notebook app are the most useful ones if you are also looking for a visualization of the forecast. In your Dynatrace tenant, open the dashboard app and create a DQL (Dynatrace Query Language) tile with the time series of a metric. You can also use my simple query below, which will show the disk usage per disk for every host in my environment. The disk usage is aggregated to an average value every two hours (see the interval).

At the bottom of the data configuration of my tile, I can find a button with Davis AI. I can activate the Davis analyzer and pick Prediction Forecast as my analyzer.

Let’s assume that I want to predict the disk usage of every disk in the environment for the upcoming 7 days. As I am measuring disk usage in a 2 hour interval, new disk usage datapoints will also be predicted in a 2 hour interval. That means, that I will need 84 datapoints to let the forecast cover the upcoming 7 days (7 days x 12 datapoints a day). For metrics with latency, you should choose for a forecast offset. If I forecast a metric which is based on log counters or values (how to create these with OpenPipeline will be explained in my next blogpost) which requires some computation, I usually include a forecast offset of 1 minute if my chosen time series interval is 1 minute, to prevent “incomplete datapoints” from entering the forecasting model. As that missing datapoint will also be predicted by Davis AI, I usually increase the data points to predict by 1. The maximum forecast offset which I can enter is 10. As a result, I will get a forecast for the next 7 days with predicted data points with a 2 hour interval. If my forecast fails, this is most likely due to not enough data points being available for the forecast – 14 datapoints are the minimum for a forecasting timeseries. By default, a 90% prediction interval will be added around the predicted disk usage values for each disk, which means that an upper and lower bound of the forecast is included, which makes the forecast more trustworthy. We expect 90% of the future points to lie within the prediction interval.

How are these future values predicted? Depending on the variance in the data, either a sampling forecaster or a linear extrapolation forecaster will be applied. If the variance is large, the sampling forecaster which is accounting for seasonal patterns such as the minute of the hour, the hour of the day or the day of the week, will be applied. With low variance, the linear extrapolation forecaster will be used . A more detailed overview of all underlying formulas, the variance thresholds and additional forecasting queries can be found here: Dynatrace Docs - Forecast Analysis

Ref: https://docs.dynatrace.com/docs/discover-dynatrace/platform/davis-ai/ai-models/forecast-analysis/

Why have I always been hyped about the forecasting feature? Because it can predict any metric which you want, no matter the metric type , as long as enough datapoints are available. This can be a great feature for many industries. For example, imagine the Dutch “zorgpiek”, when the new health care premium is announced and the health insurance companies are facing high traffic on their applications. Here, we could even predict the throughput/traffic or any other interesting metrics for the upcoming 4 years, if we stick to an interval of 1 value per day (there is a maximum of 600 data points which can be predicted). One of many interesting use cases to explore… Nevertheless, it can seem rather taxing to ask teams to constantly check the new predictions in their dashboards or re-run the predictive DQL queries stored in a notebook. This is where automation, and another great Dynatrace app called “workflows” comes into play. We can build a workflow which automatically executes the Davis analyzer prediction task and, based on the outcome, creates a problem in Dynatrace. The workflow will start with a fixed time or time interval trigger which we could set to every 24 hours or once per week on a specific day and time. At that moment in time, the workflow with all its tasks will be executed.

In the second part of the workflow, the predict_memory_usage task, is executed. If I want to predict memory usage for all monitored hosts for the upcoming 7 days, the configuration of the task can look as follows. In workflows, we are also allowed to manipulate the coverage probability. As a general rule: The higher the coverage probability, the larger the interval.

After this task, all the future points for all hosts in the environment are predicted. To raise a Dynatrace problems, two more workflow tasks, executing JavaScript code, are needed. The check_prediction task compares the predicted data points with a set thresholds, above which a Dynatrace problem should be created. In the source code, we can see that I have chosen a threshold of 80, meaning that only predicted values above 80% memory usage should be considered a violation and should lead in the creation of a problem during the final workflow task. In the source code we can also play futher with what is considered a violation. Should the predicted value be larger or smaller than the threshold? Should a violation be detected if only the final predicted value of the timeseries is above the threshold or if the average of the predicted array of values is above the threshold?

The final task, for which the input is shown below, creates the Dynatrace problem with any additional information which we provide in the body.

Technically, there is no need to be a JavaScript expert, to get started on workflows. Dynatrace provides many sample workflows, like this one, in their public playground environment. Just visit: Dynatrace Demo Environment

We could even create a second “simple workflow” (includes only a trigger and a non-JavaScript task ), starting with a Davis problem trigger sending notifications to a Teams channel, ServiceNow, or any other integration to make a specific team aware of the upcoming resource shortage. No more obligations to check the dashboard.

To sum up this blogpost, especially the combination of automation (with workflows) and forecasting (with DavisAI) can be a gamechanger for stakeholders and can even be very fun to set up. I really hope that this blogpost inspired you, to play around with dashboards, the Davis AI functionalities and the workflows app in your own or the Dynatrace Playground environment.

Kubernetes operators and Ansible: A match made in heaven?

Sun, 06 Jul 2025 00:00:00 +0000

The problem

We have all been there. You finally found an operator to deploy your favorite application. Unfortunately, the operator only manages the deployment not the actual configuration. In this specific case I want to manage Quay (container registry) and its organizations declaratively to embed it in the onboarding template for teams. Having a sysadmin background the task of writing a GO operator to manage organizations in Quay using the REST API seems daunting. Luckily the Operator SDK also supports Ansible.

The tool

Ref: https://sdk.operatorframework.io/

“The Operator SDK provides the tools to build, test, and package Operators.” The SDK offers three ways to build your operator: 1) Go 2) Ansible 3) Helm Charts. To solve the current problem, Ansible has my preference. Ansible has an active community that maintains a wide variety of collections including the quay_configuration collection maintained by the Red Hat Communities of Practice. It saves a lot of time to simply reuse existing collections and roles instead of writing them from scratch.

The solution

Prerequisites

The rest of the blog describes the steps to create an Ansible Operator for Quay. The following prerequisites are required:

OpenShift Cluster
Quay Instance
Access token to Quay with full administrative permissions
Container registry to store the container images

Building the operator

On your system (Linux or macOs) install the operator-sdk (see documentation).

Initialize a template project for the ansible operator and create its first api: QuayOrganization.

mkdir quay-configuration && cd quay-configuration
operator-sdk init --plugins=ansible --domain conclusionxforce.cloud
operator-sdk create api --group quay --version v1alpha1 --kind QuayOrganization --generate-playbook

Add the quay_configuration collection to the requirements.yml to make sure it is included in the controller image.
```
./requirements.yml

...
  - name: infra.quay_configuration
    version: "2.5.2"
...
```

Create the default playbook in playbooks/quayorganization.yml.

4.1 Add the tasks to retrieve the secret containing the superadmin token in Kubernetes.

---
- hosts: localhost
  gather_facts: no
  collections:
  - kubernetes.core
  - operator_sdk.util
  - infra.quay_configuration
  tasks:
  - name: Get the Quay Instance secret
    kubernetes.core.k8s_info:
      kind: Secret
      namespace: "{{ ansible_operator_meta.namespace }}"
      name: "{{ quay_secret.name }}"
    register: quay_secret_result

4.2 Define Quay Configuration task. Take a look at the value for quay_org_name. We use reuse the metadata.name field from the QuayOrganization resource.

  - name: Ensure the organization exists
    include_role:
      name: infra.quay_configuration.quay_org
    vars:
      # Connection parameters
      quay_org_host: "{{ quay_secret_result.resources[0].data.quayHost | b64decode }}"
      quay_org_token: "{{ quay_secret_result.resources[0].data.quayToken | b64decode }}"
      quay_validate_certs: "{{ quay_secret_result.data.quayValidateCerts | b64decode }}"
      # Organization name and email
      quay_org_name: "{{ ansible_operator_meta.name }}"
      quay_org_prune: "{{ quay_prune | default([]) }}" 
      quay_org_users: "{{ quay_users | default([]) }}" 
      quay_org_robots: "{{ quay_robots | default([]) }}" 
      quay_org_teams: "{{ quay_teams | default([]) }}" 
      quay_org_default_perms: "{{ quay_default_perms | default([]) }}" 
      quay_org_applications: "{{ quay_applications | default([]) }}" 
      quay_org_repositories: "{{ quay_repositories | default([]) }}"

Now it is time to adjust the generated Custom Resource Definition. The CRD defines the schema for our quayorganization resource. For the sake of the blog, I have only defined a few fields. The x-kubernetes-preserve-unknown-fields makes it possible to define other fields without including them in the CRD. In a production environment you would want to define the CRDs as precise as possible.

config/crd/bases/quay.conclusionxforce.cloud_quayorganizations.yaml
...
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: QuayOrganization is the Schema for the quayorganizations API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributordevel/        sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/   contributordevel/     sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: Spec defines the desired state of QuayOrganization
            type: object
            x-kubernetes-preserve-unknown-fields: true
            properties:
              quaySecret:
                description: QuaySecret defines a secret containing the information regarding your quay instance
                type: object
                properties:
                  name:
                    description: Name of the Kubernetes Secret containing quay information containing (quayHostquayUser,         quayPassword, quayValidateCerts)
                    type: string
...

Next step is to create a sample resource. Easy to use for testing!

config/samples/quay_v1alpha1_quayorganization.yaml

apiVersion: quay.conclusionxforce.cloud/v1alpha1
kind: QuayOrganization
metadata:
  labels:
    app.kubernetes.io/name: quay-configuration-operator
    app.kubernetes.io/managed-by: kustomize
  name: quayorganization-sample
spec:
  quaySecret:
    name: quay-secret

For native Kubernetes resources (like deployments or services) the operator will automatically delete the resources when the custom resource is deleted. This operator, however, configures Quay through the REST API, so we have to explicitly tell the operator to delete an organization in Quay upon the deletion of the resource, using a finalizer playbook.

7.1 Create the delete-organization playbook.

playbooks/delete-organization.yml
---
- name: Deleting an organization
  hosts: localhost
  become: false
  gather_facts: false
  collections:
  - kubernetes.core
  - operator_sdk.util
  - infra.quay_configuration
  tasks:
  - name: Get the Quay Instance secret
    kubernetes.core.k8s_info:
      kind: Secret
      namespace: "{{ ansible_operator_meta.namespace }}"
      name: "{{ quay_secret.name }}"
    register: quay_secret_result

  - name: Ensure the organization is deleted
    infra.quay_configuration.quay_organization:
      quay_host: "{{ quay_secret_result.resources[0].data.quayHost | b64decode }}"
      quay_token: "{{ quay_secret_result.resources[0].data.quayToken | b64decode }}"
      validate_certs: "{{ quay_secret_result.resources[0].data.quayValidateCerts | b64decode }}"
      name: "{{ ansible_operator_meta.name }}"
      state: absent

7.2 Now add the playbook as a finalizer.

watches.yaml
---
# Use the 'create api' subcommand to add watches to this file.
- version: v1alpha1
  group: quay.conclusionxforce.cloud
  kind: QuayOrganization
  playbook: playbooks/quayorganization.yml
  reconcilePeriod: 5m
  finalizer:
    name: quay.conclusionxforce.cloud
    playbook: playbooks/delete-organization.yml
# +kubebuilder:scaffold:watch

The sharp-eyed reader will notice that I also added a reconcilePeriod of 5 minutes to the resource. This will cause the operator to rerun the creation playbook every 5 minutes .

Generate and customize the cluster service version.

8.1 Generate the manifests

operator-sdk generate kustomize manifests --interactive=false

8.2 Now finalize your Cluster Service Version file and include your name, organization and your favourite logo,

config/manifests/bases/quay-configuration-operator.clusterserviceversion.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[]'
    capabilities: Basic Install
  name: quay-configuration-operator.v0.0.0
  namespace: placeholder
spec:
  apiservicedefinitions: {}
  customresourcedefinitions: {}
  description: Operator for configuring Quay Organizations
  displayName: Quay Configuration Operator
  icon:
  - base64data: <BASE64 encode picture>
    mediatype: PNG
  install:
    spec:
      deployments: null
    strategy: ""
  installModes:
  - supported: false
    type: OwnNamespace
  - supported: false
    type: SingleNamespace
  - supported: false
    type: MultiNamespace
  - supported: true
    type: AllNamespaces
  keywords:
  - quay
  - xforce
  links:
  - name: Conclusion Xforce
    url: https://conclusionxforce.cloud
  maintainers:
  - email: sven.kooiman@conclusionxforce.nl
    name: sven
  maturity: alpha
  minKubeVersion: 1.27.0
  provider:
    name: Conclusion xForce
  version: 0.0.0

In the Makefile change the IMAGE_TAG_BASE to the repository where you will be publishing your operator and change the default controller to reference this variable.
```
./Makefile

...
IMAGE_TAG_BASE ?= quay.io/<your organization>/quay-configuration-operator
IMG ?= $(IMAGE_TAG_BASE):$(VERSION)
...
```
Build and push your operator to the registry.
```
make docker-build docker-push
```
Create the OLM bundle and push to your container registry. This step is optional and for OpenShift/Operator Lifecycle Manager only. It is possible to directly deploy the operator to your local Kubernetes cluster or run it on your workstation)
```
make bundle bundle-build bundle-push
```
Run the actual operator.

12.1 Run the command below:
```
operator-sdk run bundle quay.io/<your organization>/quay-configuration-operator-bundle:v0.0.1
```
12.2 Check the operator in the OpenShift Console

Deploy your first Quay organization.

13.1 Create the secret containing the Quay token.

oc create secret generic quay-secret \
  --from-literal=quayHost=<insert QuayURL> \
  --from-literal=quayToken=<insert Quay Token> \
  --from-literal=quayValidateCerts=true

13.2 Now create your first custom resource using the sample created earlier

oc apply -f config/samples/quay_v1alpha1_quayorganization.yaml

Check the status of the resource and Quay.

14.1 Check the resource using the OpenShift CLI.
```
oc get quayorganization quayorganization-sample -o yaml
```
14.2 Check in Quay whether the new organization exists in Quay
Congratulations you have created your first Ansible based operator!

The end

Combining Ansible and operators provides the best of both worlds. You can leverage the ecosystem of Ansible to manage your application in a cloud native way. The example of the quay configuration operator needs a lot of finetuning. Testing with Molecule and more granular CRDs should be added for a production-ready operator. Hopefully the blog has inspired you to use an Ansible Operator next time you need to manage an application in Kubernetes.

Least Privileged access in ArgoCD

Tue, 24 Jun 2025 00:00:00 +0000

Introduction

In my last blog, I already mentioned the ArgoCD Ephemeral Access Extension briefly. I was very excited about it when I learned from it on Kubecon: I think it is a valuable tool to increase the security of a Kubernetes production cluster. The authors of the Ephemeral Access Controller wrote the GUI, it was/is up to the community to write plugins for several tools. It is the role of the GUI to allow the user to request permissions, the role of the plugin is to grant or deny the request. In this way, the same Access Extension can deal with different reasons to allow or deny the request. I wrote the plugin for ServiceNow, you can find it on my Github repository [1]. Information how to install and configure the plugin is in that repository as well. In this blog I will describe the plugin from an operator point of view.

How does it work?

When the user requests more access, the Ephemeral Access Extension will call the plugin to do the checks. The plugin should be written in go. The request comes in on the GrantAccess method, which will do the tests. When all tests pass, then the plugin can return plugin.GrantStatusGranted in the status field, to indicate that the request is granted. Likewise, plugin.GrantStatusDenied can be returned when the request is denied. When the plugin needs more time, then plugin.GrantStatusPending can be returned, the Ephemeral Access Extension will then call on GrantAccess later again to see if a new status is available. My plugin only returns GrantStatusGranted or GrantStatusDenied, to keep it simple.

When you want to get a deep dive of the Ephemeral Access Extension itself, you can read the information on the Github repository [2].

Checks on the Change Item

When the plugin is called, it will first look in the lables of the ArgoCD application which Configuration Item (CI) in the CMDB belongs to this application. In this example, I will use the default ciName as label key and app-demoapp as name of the CI.

The plugin will call the API of ServiceNow and get the information of the CI. Based on that, it decides if the CI is valid. There are multiple values in ServiceNow for a CI. The plugin will allow a request for a CI that has one of the following statuses:

1 = Installed
3 = In maintenance
4 = Pending install
5 = Pending repair

When the status is different, then the request is denied. It doesn’t make sense to change an application that has a state of Absent, Retired or Stolen.

The username and password for the ServiceNow API are stored in a secret in Kubernetes, the URL of the ServiceNow API is stored in environment variables of the controller deployment in the argocd-ephemeral-access namespace.

Checks on the changes

When the CI is valid, then the changes are retrieved. It is currently not possible to add a change number to the request from ArgoCD, this means that we have to get all changes that might be relevant to our request. This has two big disadvantates: when there are lots of changes, then the request to the ServiceNow API might become very slow. There might also be multiple valid changes, currently the plugin will take the first one that is sent by the API. This might not be the correct one, and this might impact the amount of time that one gets to do the change.

To limit the number of changes that are sent back by the API, most of the checks on the change are done by the API. For example: we only want to get back changes that are approved. The full list of checks is:

Is the change connected to the correct and valid CI (see before)?
Does the change have the state “Implement”?
Does the change have the phase “Requested”?
Is the change approved?
Is it an “active” change?
Does the change not start before 1 week back?
Does the change not end after 1 week from now?

These last two checks are meant to limit the amount of changes that we get from the API and I hope that this will speed up the calls to the API in a bigger production environment.

Duration of the extra permissions

Currently, the Ephemeral Access Extension passes the duration of the permission in the Access Request to the plugin. Unfortunately there doesn’t seem to be a way to change this duration: not from the go code in the plugin, not by changing the AccessRequest in Kubernetes. This means, that by default a request is either granted for the full duration of the request (which is by default 4 hours), or it is denied. I think that it would be wise to change this and let the plugin decide the duration of the permissions. From an ITIL point of view, changes have different durations. A change can be two hours, four hours, one day - maybe even multiple days when the change is part of a very big change where the work will take a whole weekend.

The question is: how much time should the permission be granted? I raised this as an issue on the Ephemeral Access Extension github repository. The conclusion was that the plugin should be able to limit the duration, but not extend the default duration. This will be implemented in a future release of the Ephemeral Access Extension.

Up to the moment that this change is implemented, the plugin has to work around this issue. There is one (quite dirty) trick around this, and that is to delete the AccessRequest in Kubernetes. The effect is that permissions are revoked on the moment that the AccessRequest is no longer there. For the ServiceNow plugin this means that:

When a change has an end date that is later than the duration in the original request, then the Ephemeral Access Extension will stop the permission after this duration,
When a change has an end date that is earlier the duration in the original request, then a Kubernetes CronJob is created that will delete the AccessRequest. To prevent CronJobs to be kept forever, the job that is started by the CronJob will remove the CronJob after the AccessRequest is deleted. Unfortunately, this means that the logs are no longer present after this. Unfortunately there doesn’t seem to be a way to start jobs only once on a certain time. It would be nice to have such a feature (and then use ttlSecondsAfterFinished in that type of Job as well to clean them up automatically).

Reporting back to the operator

When the AccessRequest is granted, I used Markdown language to make fields like the change number and the end date and the duration bold. This is important because the difference between the date and time in the message are leading, not the time in the “Expires” field that is shown by the Ephemeral Access Extension.

Unfortunately, the Ephemeral Access Extension doesn’t give a nice message when the request is denied. One has to search in the AccessRequests within Kubernetes or in the logs of the plugin to find out why a request is denied. The people who maintain the Ephemeral Access Extension will fix this in the future by adding a history of requests, see this issue.

When the request is granted, the ServiceNow plugin will also write a note in the change that the request is granted. In this way, everyone who is interested in the change can see who was granted access and for how long.

Exclusion roles

I think that for many environments this functionality is enough to work on a day-by-day base. There might, however, come a day where a lot goes wrong. The day that you don’t have time to create a change, keep the duration and the statuses in ServiceNow correct etcetera. Maybe ServiceNow itself has API errors and changes cannot be checked. For these use cases, I created “Exclusion Roles”: roles of the Ephemeral Access Controller that will not look at CIs and changes, but will immediately grant access.

You might use this for incident managers, who coordinate an incident and add administrators to the incidentmanager group on the moment that these issues happen. Or you might have roles of “Senior Staff” that knows when to bypass Service Now, and when not to do so. It shouldn’t become a “normal way of working” to use this mechanism, the plugin will give a warning when someone gets access via an exclusion role:

when someone is granted permission based on a change, this information has INFO severity:

You can see here that the timezone on the server is different from the timezone in the UI. This isn’t an issue. It is, however, important to use the same timezone in the environment variables of the plugin and the timezone of the user in the ServiceNow environment that the plugin is using.

Changes in existing Kubernetes resources

The username and password for the ServiceNow API are stored in a secret in Kubernetes. The ClusterRole controller-role (that is created by the Ephemeral Access Extension) has to be changed to be able to read the secret. The ClusterRole should also be allowed to create CronJobs.

The controller deployment has to be changed as well: I added a lot of environment variables, some of them use defaults. The “value at deployment” is the value that you can find in the Github repository under manifests > plugin > controller-patch.yaml. When you remove the environment variable, sometimes it will use a sensible default. For other environment variables the plugin will deny permissions when the environment variable is not present.

Environment variable	Value at deployment	Default value	Remarks
SECRET_NAMESPACE	argocd-ephemeral-access	argocd-ephemeral-access
SERVICENOW_SECRET_NAME	CHANGE_THIS_WITH_THE_SERVICENOW_SECRET_NAME	servicenow-secret	(1) (2)
SERVICENOW_URL	CHANGE_THIS_TO_POINT_TO_YOUR_SERVICENOW_URL	n/a	(1) (3)
EXCLUSION_ROLES	incidentmanager	n/a	(4)
TIMEZONE	Europe/Amsterdam	UTC	(2)
CI_LABEL	ciName	ciName
EPHEMERAL_PLUGIN_PATH	/tmp/plugin/plugin	n/a	(5)

(1) Plugin will deny access when this value is not changed
(2) When this environment variable is removed, the value under “Default value” is used
(3) When this environment variable is removed, the plugin will deny access
(4) When you want to use incidentmanager as an exception role, you need to create AccessBindings and RoleTemplates for incidentmanager as well. You can look in the cloudformation directory for a working environment in AWS for this configuration.
(5) Don’t change this, it is part of the Ephemeral Access Extension configuration.

How to create a ServiceNow development environment

One of the advantages of ServiceNow is that everyone can create a development environment in minutes. You need to register a developer account [3] and then you can create a ServiceNow instance, including access to all APIs. When you don’t use the ServiceNow instance for 10 days, then the instance will be deleted.

There are two ways of configuring the development environment: via a script and via the ServiceNow portal. To get the URL, the username and the password for your instance, log in to the developer platform and go to your profile (initials in the upper-right corner of the screen). Click under Instance Action on menu item “Manage instance password”:

You now have the Instance URL, that can be used both for the API and for the portal.

Via a script

Not all settings can be changed using a script. When you want to create a dedicated plugin user in ServiceNow, you still have to do this via the portal. Please mind to set the timezone of this plugin user to be the same as the environment parameter of the plugin. It is possible to use the admin user as the user for the plugin as well, which can be useful for testing the plugin.

I added two scripts to the repository: one to create a CI class with a CI, the other one to add a change with the valid statuses. The scripts are in the /examples/servicenow-scripts directory. Both scripts need the ServiceNow URL, the username and the password as parameters, for example:

create-ci-class-with-ci.sh https://dev123456.service-now.com admin a%B12CdE
create-change.sh https://dev123456.service-now.com admin a%B12CdE

When you want to know more about the ideas behind the script, I will explain that in the next part about the configuration via the ServiceNow portal.

Via the ServiceNow portal

Change the timezone

First change the timezone in the console: click in the upper right corner on your account (for admin users: the image of the smiling guy), then select Profile. Change the Time zone (in my case: to Europe/Amsterdam) and click on the Update button.

Create a CI

For our solution, we need (at least) one Configuration Item (CI) and one Change. CIs in ServiceNow are part of CI groups. We could use the already existing group Application, but it might be better to create a new group Kubernetes Application. Use the hamburger menu on the upper left and then search for CI Class Manager.

Click on the button Hierarchy.

Hover over the Configuration Item item in the menu and click on the hamburger menu.

Click in the menu on Add child class.

The Display Name is mandatory and is the base for the internal table name within ServiceNow:

The rest of the fields are not required, click 4 times on “Next”, then “Done”.

In the left menu, click on CI List, then on New. Give your application a name, for example app-demoapp. Click then on Submit.

We now have a Kubernetes application in the correct the Installed state. You can change this if you like to see how the plugin behaves on different statuses.

Create a change

When you want to use the console to create a change, use the hamburger menu in the top menu and search for change. Select the menu item Create new. After that, select the menu option Models. You can see here the options to create an emergency change or a normal change. Select a Normal change for now.

Change the field Configuration item to app-demoapp and select tab Schedule to change the start and end date of the change. Use for example a timeframe from more-or-less now to a few hours later. Click on Submit after that, and go in the main hamburger menu to Change > Open.

You can find your change in this list, when you follow along in a new developer environment this change will have the number CHG0030001.

Open the change, and then change the State to Assess. Click on the magnifier glass next to Assignment Group to search for a group, for example Application Development. After that, click on the “Update” button on the top of the screen.

You will return to the list of changes, open the same change again. The change is now waiting for approval: select the “Approvers (5)” tab on the lower part of the screen and hover over the first name. Click on the check box on the left to let (in my case) Arya Hajarha approve your change.

Use the “Actions on selected rows” to approve the change.

The state is changing to state Authorize, it now has to be approved by someone of the CAB Approval group. Follow the same procedure again to let the first member of the CAB Approval group to approve your change.

Now change the state from “Scheduled” to “Implement” and click in the top of the screen on Update. Now your change is in the correct state for the Ephemeral Access Controller Plugin to allow extra permissions.

Links

[1] Github repository ServiceNow Plugin: https://github.com/argoproj-labs/argocd-ephemeral-access-plugin-servicenow [2] Github repository Ephemeral Access Extension: https://github.com/argoproj-labs/argocd-ephemeral-access
[3] Developer environment ServiceNow: https://developer.servicenow.com/

Running a RHEL-based WSL

Thu, 29 May 2025 00:00:00 +0000

Introduction

This blog explains how you can create your own WSL image based on Red Hat Enterprise Linux, and why you would want to use WSL in the first place.

What is WSL?

The Windows Subsystem for Linux (which could be considered to be a Linux subsystem for Windows) is a Windows feature which allows you to run a Linux distribution within your Windows setup. It uses a Microsoft-provided Linux kernel and tight integration with the Windows host operating system. Microsoft offers several distributions via its Marketplace, but Red Hat Enterprise Linux is not one of them.

You can start your Linux distro from the Windows command line (cmd or powershell) by typing wsl --distribution <DistroName>. Next you’ll end up in the login shell of your Distro. WSL has created a NAT interface for you so your Linux distro has network connectivity to the outside world. All Windows drives are mounted as /mnt/<DriveLetter>.

Why would you need such a WSL? Suppose you are writing ansible playbooks. You can use VS Code, as it has nice code editing features and an ansible and yaml extension. However in order to use the ansible-lint integration, you need to be able to run ansible - but it’s not possible to run ansible on Windows. Now VS Code has WSL integration built in via a “remote development” extension, allowing you to edit your code inside the WSL image and thus providing access to any tool present in the WSL image, like ansible-lint etc.

So write your ansible code inside your WSL image using the Remote - WSL extension, save the file and see ansible-lint directly verify your file in the VS Code editor window! And if you would want to run the code with ansible-navigator, you could even use the same Execution Environment in this WSL that is meant to be used in your Ansible Automation Platform setup.

What about the RHEL WSL?

As said, Microsoft offers several WSL images via its Marketplace, but not RHEL. If you would want to have a RHEL WSL, you were quite on your own with maybe a bit of help from Google. But not anymore: at Red Hat Summit 2025 it’s officially announced that you can create WSL images for RHEL 8, 9 and 10 using Red Hat’s image builder tool. The downside: there’s still no RHEL image in the Microsoft Marketplace (but there is now a Fedora 42 image available). However using the image builder you can really tweak the image to suit your needs. At the same time Red Hat offers WSL2 images in the download section of Red Hat Enterprise Linux. Obviously, as it is RHEL, you will need a RHEL subscription in order to use the RHEL WSL image. A developer subscription (free to use, valid for 1 year, available from Red Hat) will work just fine.

Red Hat Image Builder is available through the Red Hat Hybrid Cloud Console Red Hat customer portal, in the Red hat Enterprise Linux section (“Insights for RHEL”). Start the image builder by selecting Inventory, Images: .

In order to create images with Image Builder, you need to define the blueprint for the image. In this blueprint you specify which RHEL version (8, 9 or 10) and which architecture (x86_64 or aarch64) should be used for the image, and you define the target environment, which can be public cloud (e.g. AWS or Azure), some form of virtualization or … Windows Subsystem for Linux (WSL).

At the time of writing, RHEL-10 has just been released and Ansible Automation Platform is not yet available on RHEL-10. So if you want to build a WSL for Ansible development, use RHEL-9 for now.

In the optional steps of the image creation, you can insert one of your own activation keys so the WSL is automatically registered at Red Hat (Image Builder will attach it to your organization). You can further tweak the WSL image blueprint to your liking, but leave the OpenSCAP and Filesystem configuration on its defaults. A new feature in Image Builder is the option to directly create a user account within the image. This is especially useful for the WSL image, as otherwise you would always work as ‘root’ in the WSL. When all configuration is done, the blueprint can be saved and the WSL image can be created. The result is a tarball (.tar.gz) which is ready for download for a limited period of time.

Creating the WSL

Once the WSL image is created, it’s time to create the actual WSL instance on your Windows host. When the image file is downloaded on the Windows host, it can be imported into WSL with wsl --import <DistroName> <DestinationDir> <FileName> where:

DistroName is the name of the distribution in your WSL setup
DestinationDir is the path where the WSL should have its persistent storage
FileName is the name of the WSL image file

You can directly specify the compressed tar file (.tar.gz); there is no need to first decompress. Next you can start the WSL with wsl -d <DistroName>. If you created a user account during the image creation process, you can directly connect to the wsl as that user with wsl -d <DistroName> -u <UserName>. If you always want to connect as that user, you can make it the default user with wsl --manage <DistroName> --set-default-user <UserName>.

ArgoCD Single Sign On based on AWS Cognito User Pools

Wed, 21 May 2025 00:00:00 +0000

Introduction

In this blog, I will show how you can connect ArgoCD and AWS Cognito via OIDC. It took me some time to figure out how to configure AWS CloudFormation and how to let AWS Cognito and ArgoCD work together. You can use my GitHub repository for a reference when you are struggling yourself [1].

AWS Cognito

AWS Cognito is an identity provider based on OIDC. It can create, update and delete users and groups in user pools and store application data. We need the application data for example to get secrets that give read permissions for the users and groups in the user pool. Cognito can also connect to other identity providers like Facebook, Google, Amazon, X and any OIDC or SAML provider. You can also configure guest access if you need to.

In this example I will keep it simple: I will create one user in one administrator group in an AWS Cognito user pool and then connect that user pool to ArgoCD.

ArgoCD

ArgoCD is a tool to connect Git repositories to Kubernetes. Any change in the Git repository will be deployed in the Kubernetes environment, this can be done both manually and automatically. ArgoCD can also be configured to revert drift: when a Kubernetes environment changes and the Git repository stays the same, ArgoCD can revert the changes in the live Kubernetes environment. ArgoCD is used in about half of the Kubernetes environments according to the maintainers of ArgoCD.

There are two default roles in ArgoCD: readonly and administrator. In this demo my user in the AWS Cognito User Pool will be a read-only user by default.

ArgoCD Ephemeral Access Extension

In many cases, just using ArgoCD is enough for your environment. Sometimes, however, you want to have more control on the ArgoCD application. This might be the case when you want to synchronize only with the Git repository when there is an approved change in, for example, ServiceNow.

The ArgoCD part of this solution is already available: it is called the Ephemeral Access Extension. In the ArgoCD user interface you can request permissions that are assigned to one of the roles for your user group. When access is granted then you will have elevated privileges in ArgoCD for a limited time. The permissions, the names of the roles, the number of roles, the text that is shown when you select the role (and more) are all configurable. When the permission is granted (and without an extra plugin you will always be granted the permission), then an ArgoCD role is attached to the ArgoCD Project. For more information please look at the Github project of the Ephemeral Access Extension [2].

In this example I will use the Ephemeral Access Extension to show how OIDC works within ArgoCD. The Ephemeral Access Extension will give more permissions than just read-only when these permissions are requested. In my example one can request either DevOps permissions (that allows for synchronization but not for deletion of resources) or administrator (where one has all permissions within ArgoCD).

OIDC configuration within ArgoCD

There are two places where OIDC is configured: it is configured in the config map argocd-cm (which describes the connection with AWS Cognito) and it is also configured in the config map argocd-rbac-cm (which describes the default permissions of the Cognito groups in ArgoCD).

The argocd-cm config map looks like:

The name is the name that you will see in the login screen of ArgoCD. It is not used in the communication with AWS Cognito.

The issuer is the first part of the Token signing key URL in AWS Cognito: you have to skip the “.well-known/jwks.json” part.

You can find the client ID and the client secret by going to the App clients submenu in the Cognito User Pool.

It is not possible to add the groups to the requestedScopes in AWS Cognito. To get the groups we have to use the requestedIDTokenClaims setting. More information about the way this should be configured for other OIDC providers can be found in the ArgoCD documentation [3].

The redirectUrl is your ArgoCD URL with prefix /auth/callback. Don’t forget to add the url-part just below the oidc.config definition. The information about the groups are retrieved using the GetUserInfo API call. This is configured by enabling the getUserInfo parameter. When you forget to do so, you will get the error Failed to query provider “ISSUER”: Get “ISSUER/.well-known/openid-configuration”: unsupported protocol scheme “” when you click the Cognito button in the logon screen of ArgoCD.

Effects in ArgoCD Ephemeral Access Extension

All people of an OIDC group can request a role when this is configured within the Ephemeral Access Extension, by clicking on the permission button. When the permissions are granted, the role is assigned to just the person who requests the role. This is done by assigning the role to the email address, not to the group.

To connect the email address of the user to his OpenID, the email address should be added to the scope of the RBAC configuration:

By doing this, you will see that a user has now an extra group in ArgoCD: the groups that were assigned within AWS Cognito and ones own email address.

AWS CloudFormation

AWS CloudFormation can be used to deploy AWS resources using CloudFormation templates.

In this example I’m using AWS CloudFormation to deploy an AWS Cognito User Pool with one user in one group. The template also deploys three EC2 nodes. I’m using cfn-init to put the configuration files in the /opt/xforce directory on these nodes. Data from different AWS resources is injected in these configuration files by a configuration script. This script also installs ArgoCD and the ArgoCD Ephemeral Access Extension.

You can use this CloudFormation script yourself to deploy this example[1] to your own AWS environment. To use it, you first have to create an empty AWS S3 bucket with the name <consultant-name>-<profile-name> (f.e. frederique-xforce-sandbox1). Change the variables in the start-k8s.sh and stop-k8s.sh scripts as well. You also need an certificate ID from AWS Certificate Manager to make ArgoCD access via https possible. In my case I used a star certificate for *.sandbox1.prutsforce.nl.

Configuring AWS Cognito in CloudFormation

It took me quite some time to configure AWS Cognito in the AWS CloudFormation template: when I tested the login page via User pool > App client > View login page, I got an error message “Login pages unavailable, Please contact an administrator”.

After a few hours I discovered that I forgot to add a AWS::Cognito::ManagedLoginBranding resource. When I added this resource, the login page started working.

My definition of the user pool, user pool client and login branding resources are:

Conclusion

It can take some time to configure OIDC in an application. In this example I showed how to use AWS Cognito as an OIDC provider for ArgoCD. I also wrote a CloudFormation template to get a working environment. I hope you enjoyed reading this blog as much as I enjoyed writing it!

Links

[1] Github repo: https://github.com/FrederiqueRetsema/ArgoCD-SSO-based-on-AWS-Cognito-Userpools
[2] Ephemeral Access Extension repo: https://github.com/argoproj-labs/argocd-ephemeral-access
[3] ArgoCD documentation about OIDC: https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/

Photo by Growtika on Unsplash

Advanced Observability for LLMs

Fri, 25 Apr 2025 00:00:00 +0000

Observability has been a hot topic for quite some time. Even though most organizations are still in the process of adopting standardized processes and platforms for gathering, shipping, and analyzing observability data (logs, metrics and traces) across their IT environments, a new challenge is approaching fast: generative AI.

Traditional applications are specific, deterministic and have predictable performance characteristics. Large Language Models (LLMs), however, are a paradigm shift. LLMs are black boxes, capable of tasks ranging from summarizing text, generating code, answering questions and more. In short: they are incredibly powerful, but opaque. As it is extremely easy to leverage the power of LLMs from traditional applications through standardized APIs, more and more applications include LLM-powered features. So, measuring performance, context, input, output, and cost when using LLMs is paramount.

Open technology to the rescue

OpenTelemetry (OTEL) is a collection of APIs, SDKs, and tools, to instrument, generate, collect, and export telemetry data. It has emerged as the standard solution for observability, as it is 100% free and open source, vendor-neutral and adopted and supported by all observability leaders. OpenLLMetry (open source, developed and maintained by Traceloop ) builds on top of OTEL, and makes LLM-powered workloads a first-class citizen in the world of observability.

I created a chatbot with Streamlit, vLLM and DeepSeek-R1-Distill-Qwen running on a server in the Scaleway cloud with an NVIDIA accelerator to illustrate the combined power of OTEL, OpenLLMetry, and Dynatrace:

Demo design

My vLLM dashboard

Prompts and responses blurred to protect the innocent

Distributed tracing, requests flow from frontend to LLM

As you can see, every call to my LLM is available in Dynatrace, including all chat context and the amount of input and output tokens. As I have also instrumented the vLLM model server, more detailed performance metrics are available for each incoming request, such as time spent in queues and time-to-first-token. The Dynatrace OneAgent is running on the server, so we have access to hardware metrics too, giving real-time insight into GPU power usage and utilization, enabling us to determine the amount of energy used per request to our model!

In conclusion: as the growth of LLM-powered workloads is increasing, we need to step up our observability game. Open standards and technologies like OTEL and OpenLLMetry are ready to solve tomorrow’s observability challenges today. If you want to see the demo in real life and step up your LLM observability game, please get in touch!

Note: This post originally appeared on LinkedIn